class_docstr: Implementation of the AMSGrad optimization algorithm.
  
  See: `On the Convergence of Adam and Beyond - [Reddi et al., 2018] <https://openreview.net/pdf?id=ryQu7f-RZ>`__.
  
  Parameters
  ----------
  learning_rate: float
      A Tensor or a floating point value.  The learning rate.
  beta1: float
      A float value or a constant float tensor.
      The exponential decay rate for the 1st moment estimates.
  beta2: float
      A float value or a constant float tensor.
      The exponential decay rate for the 2nd moment estimates.
  epsilon: float
      A small constant for numerical stability.
      This epsilon is "epsilon hat" in the Kingma and Ba paper
      (in the formula just before Section 2.1), not the epsilon in Algorithm 1 of the paper.
  use_locking: bool
      If True use locks for update operations.
  name: str
      Optional name for the operations created when applying gradients.
      Defaults to "AMSGrad".
code: def _apply_dense(self, grad, var):
      beta1_power = math_ops.cast(
          self._beta1_power, var.dtype.base_dtype
      )
      beta2_power = math_ops.cast(
          self._beta2_power, var.dtype.base_dtype
      )
      lr_t = math_ops.cast(self._lr_t, var.dtype.base_dtype)
      beta1_t = math_ops.cast(self._beta1_t, var.dtype.base_dtype)
      beta2_t = math_ops.cast(self._beta2_t, var.dtype.base_dtype)
      epsilon_t = math_ops.cast(self._epsilon_t, var.dtype.base_dtype)
  
      lr = lr_t * math_ops.sqrt(1 - beta2_power) / (1 - beta1_power)
  
      # m_t = beta1 * m + (1 - beta1) * g_t
      m = self.get_slot(var, "m")
      m_scaled_g_values = grad * (1 - beta1_t)
      m_t = state_ops.assign(
          m,
          beta1_t * m + m_scaled_g_values,
          use_locking=self._use_locking,
      )
  
      # v_t = beta2 * v + (1 - beta2) * (g_t * g_t)
      v = self.get_slot(var, "v")
      v_scaled_g_values = (grad * grad) * (1 - beta2_t)
      v_t = state_ops.assign(
          v,
          beta2_t * v + v_scaled_g_values,
          use_locking=self._use_locking,
      )
  
      # amsgrad
      vhat = self.get_slot(var, "vhat")
      vhat_t = state_ops.assign(vhat, math_ops.maximum(v_t, vhat))
      v_sqrt = math_ops.sqrt(vhat_t)
  
      var_update = state_ops.assign_sub(
          var,
          lr * m_t / (v_sqrt + epsilon_t),
          use_locking=self._use_locking,
      )
      return control_flow_ops.group(*[var_update, m_t, v_t, vhat_t])
  
file_path: TensorLayer/tensorlayer/optimizers/amsgrad.py
imports: from tensorflow.python.eager import context
  from tensorflow.python.framework import ops
  from tensorflow.python.ops import control_flow_ops, math_ops, resource_variable_ops, state_ops, variable_scope
  from tensorflow.python.training import optimizer
label_desc: Use image node_id 4 for calling the AMSGrad obj's underlying member method code with example usage: obj._apply_dense(grad, var) and returns: control_flow_ops