class_docstr: Optimizer that implements the YellowFin algorithm.
  
  See [Zhang et. al., 2017](https://arxiv.org/abs/1706.03471) for details.
docstr: Construct a new YellowFin optimizer.
  
  Implemented as a wrapper around tf.train.MomentumOptimizer
  
  Args:
    learning_rate: A Tensor or a floating point value.  The learning rate.
      Set to 1.0 in the paper.
    momentum: A Tensor or a floating point value.  The momentum.
       Set to 0.0 in the paper.
    clip_thresh: A Tensor or a floating point value. The clipping threshold
      for `tf.clip_by_global_norm`.  If None, no clipping will be carried out.
    beta: A float value or a constant float tensor.  The smoothing parameter
      for estimations.
    curvature_window_width: A int value or a constant int tensor.
      The curvature window width.
    zero_debias: A boolean, zero debias moving-averages.
    delta_mu: For extensions. Not necessary in the basic use.
    sparsity_debias: A boolean. Gradient norm and curvature are
      biased to larger values when calculated with sparse gradient.
      This is useful when the model is very sparse, e.g. LSTM with
      word embedding. For non-sparse CNN, turning it off could
      slightly accelerate the speed.
    use_locking: If True, use locks for update operations.
    name: Optional name prefix for the operations created when
      applying gradients. Defaults to "YellowFin".
    use_nesterov: If True, the underlying MomentumOptimizer uses Nesterov
      Momentum. Set to False in the default YellowFin algorithm.
  
  Note:
    clip_thresh is the threshold value on ||lr * gradient||,
    delta_mu can be place holder/variable/tensor scalar.
    They are used for additional momentum in situations such as
    asynchronous-parallel training.
    The default is 0.0(or None) for basic usage of the optimizer.
  
  Other features:
    If you want to manually control the learning rates, self.lr_factor is
    an interface to the outside, it is an multiplier for the internal
    learning rate in YellowFin. It is helpful when you want to do additional
    hand tuning or some decaying scheme to the tuned learning rate in
    YellowFin.
    Example on using lr_factor can be found here:
    https://github.com/JianGoForIt/YellowFin/blob/master/char-rnn-tensorflow/train_YF.py#L140
docstr_tok: ["Construct","a","new","YellowFin","optimizer",".","Implemented","as","a","wrapper","around","tf.train.MomentumOptimizer","Args",":","learning_rate",":","A","Tensor","or","a","floating","point","value",".","The","learning","rate",".","Set","to","1.0","in","the","paper",".","momentum",":","A","Tensor","or","a","floating","point","value",".","The","momentum",".","Set","to","0.0","in","the","paper",".","clip_thresh",":","A","Tensor","or","a","floating","point","value",".","The","clipping","threshold","for","`","tf.clip_by_global_norm","`",".","If","None",",","no","clipping","will","be","carried","out",".","beta",":","A","float","value","or","a","constant","float","tensor",".","The","smoothing","parameter","for","estimations",".","curvature_window_width",":","A","int","value","or","a","constant","int","tensor",".","The","curvature","window","width",".","zero_debias",":","A","boolean",",","zero","debias","moving-averages",".","delta_mu",":","For","extensions",".","Not","necessary","in","the","basic","use",".","sparsity_debias",":","A","boolean",".","Gradient","norm","and","curvature","are","biased","to","larger","values","when","calculated","with","sparse","gradient",".","This","is","useful","when","the","model","is","very","sparse",",","e.g",".","LSTM","with","word","embedding",".","For","non-sparse","CNN",",","turning","it","off","could","slightly","accelerate","the","speed",".","use_locking",":","If","True",",","use","locks","for","update","operations",".","name",":","Optional","name","prefix","for","the","operations","created","when","applying","gradients",".","Defaults","to","``","YellowFin","''",".","use_nesterov",":","If","True",",","the","underlying","MomentumOptimizer","uses","Nesterov","Momentum",".","Set","to","False","in","the","default","YellowFin","algorithm",".","Note",":","clip_thresh","is","the","threshold","value","on","||lr","*","gradient||",",","delta_mu","can","be","place","holder\/variable\/tensor","scalar",".","They","are","used","for","additional","momentum","in","situations","such","as","asynchronous-parallel","training",".","The","default","is","0.0","(","or","None",")","for","basic","usage","of","the","optimizer",".","Other","features",":","If","you","want","to","manually","control","the","learning","rates",",","self.lr_factor","is","an","interface","to","the","outside",",","it","is","an","multiplier","for","the","internal","learning","rate","in","YellowFin",".","It","is","helpful","when","you","want","to","do","additional","hand","tuning","or","some","decaying","scheme","to","the","tuned","learning","rate","in","YellowFin",".","Example","on","using","lr_factor","can","be","found","here",":","https",":","\/\/github.com\/JianGoForIt\/YellowFin\/blob\/master\/char-rnn-tensorflow\/train_YF.py","#","L140"]
code: def __init__(
      self,
      learning_rate=1.0,
      momentum=0.0,
      clip_thresh=None,
      beta=0.999,
      curvature_window_width=20,
      zero_debias=True,
      delta_mu=0.0,
      sparsity_debias=True,
      use_locking=False,
      name="YellowFin",
      use_nesterov=False,
  ):
      """Construct a new YellowFin optimizer.
  
      Implemented as a wrapper around tf.train.MomentumOptimizer
  
      Args:
        learning_rate: A Tensor or a floating point value.  The learning rate.
          Set to 1.0 in the paper.
        momentum: A Tensor or a floating point value.  The momentum.
           Set to 0.0 in the paper.
        clip_thresh: A Tensor or a floating point value. The clipping threshold
          for `tf.clip_by_global_norm`.  If None, no clipping will be carried out.
        beta: A float value or a constant float tensor.  The smoothing parameter
          for estimations.
        curvature_window_width: A int value or a constant int tensor.
          The curvature window width.
        zero_debias: A boolean, zero debias moving-averages.
        delta_mu: For extensions. Not necessary in the basic use.
        sparsity_debias: A boolean. Gradient norm and curvature are
          biased to larger values when calculated with sparse gradient.
          This is useful when the model is very sparse, e.g. LSTM with
          word embedding. For non-sparse CNN, turning it off could
          slightly accelerate the speed.
        use_locking: If True, use locks for update operations.
        name: Optional name prefix for the operations created when
          applying gradients. Defaults to "YellowFin".
        use_nesterov: If True, the underlying MomentumOptimizer uses Nesterov
          Momentum. Set to False in the default YellowFin algorithm.
  
      Note:
        clip_thresh is the threshold value on ||lr * gradient||,
        delta_mu can be place holder/variable/tensor scalar.
        They are used for additional momentum in situations such as
        asynchronous-parallel training.
        The default is 0.0(or None) for basic usage of the optimizer.
  
      Other features:
        If you want to manually control the learning rates, self.lr_factor is
        an interface to the outside, it is an multiplier for the internal
        learning rate in YellowFin. It is helpful when you want to do additional
        hand tuning or some decaying scheme to the tuned learning rate in
        YellowFin.
        Example on using lr_factor can be found here:
        https://github.com/JianGoForIt/YellowFin/blob/master/char-rnn-tensorflow/train_YF.py#L140
      """
      # Set lr and mu
      self._lr = learning_rate
      self._mu = momentum
  
      # Set lr and mu tensor.
      self._lr_var = tf.get_variable(
          "YF_lr",
          dtype=tf.float32,
          trainable=False,
          initializer=learning_rate,
      )
      self._mu_var = tf.get_variable(
          "YF_mu",
          dtype=tf.float32,
          trainable=False,
          initializer=tf.constant(momentum),
      )
  
      # Tuning factor for learning rates step or decaying scheme.
      self.lr_factor = tf.get_variable(
          "YF_lr_factor",
          dtype=tf.float32,
          trainable=False,
          initializer=tf.constant(1.0),
      )
  
      # Gradient Clipping Threshold.
      if clip_thresh is not None:
          self._clip_thresh_var = tf.get_variable(
              "YF_clip_thresh",
              dtype=tf.float32,
              trainable=False,
              initializer=tf.constant(clip_thresh),
          )
      else:
          self._clip_thresh_var = None
  
      # Set initial lr and mu for momentum.
      self._lr_m = self._lr_var * self.lr_factor
      self._mu_m = self._mu_var + delta_mu
  
      # Init momentum optimizer.
      self._momentum_optimizer = tf.train.MomentumOptimizer(
          self._lr_m, self._mu_m, use_locking, name, use_nesterov
      )
  
      # Moving average for statistics.
      self._beta = beta
      self._moving_averager = None
  
      # Step counting.
      self._step = tf.get_variable(
          "YF_step",
          dtype=tf.int32,
          trainable=False,
          initializer=tf.constant(0),
      )
  
      # YF_step + 1 op.
      self._increment_step_op = None
  
      # For conditional tuning.
      self._do_tune = tf.greater(self._step, tf.constant(0))
  
      # Moving-averages.
      self._zero_debias = zero_debias
      self._sparsity_debias = sparsity_debias
  
      # For curvature range.
      self.curvature_window_width = curvature_window_width
      self._curv_win = None
  
      # Gradients and Variables.
      self._grad = None
      self._vars = None
  
      # Get per var g**2, norm**2 and mean(norm**2).
      self._grad_squared = None
      self._grad_norm_squared = None
      self._grad_norm_squared_avg = None
  
      # Mean(grad) and Mean(grad**2) to compute Variance.
      self._grad_avg = None
      self._grad_avg_squared = None
  
      # Max and Min curvature variations.
      self._h_max_t = None
      self._h_min_t = None
      self._h_min = None
      self._h_max = None
  
      # Gradient Expected Variance.
      self._grad_var = None
  
      # Gradient Norm and Mean(Gradient Norm).
      self._grad_norm = None
      self._grad_norm_avg = None
  
      # Distance to optimum and Mean(Distance to optimum).
      self._d_t = None
      self._dist_to_opt_avg = None
  
      # Maintains moving averages of variables
      # by employing an exponential decay(Beta),
      # and (zero_devias) moving-averages.
      self._moving_averager = None
  
      # Handling Sparse Matrix
      self._sparsity = None
      self._sparsity_avg = None
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import tensorflow.compat.v1
label_desc: Use image node_id 1 to create a new YellowFinOptimizer object from inherited base classes: object with example: obj = YellowFinOptimizer(learning_rate, momentum, clip_thresh, beta, curvature_window_width, zero_debias, delta_mu, sparsity_debias, use_locking, name, use_nesterov)