class_docstr: TrainOp.
  
  TrainOp represents a set of operation used for optimizing a network.
  
  A TrainOp is meant to hold all training parameters of an optimizer.
  `Trainer` class will then instantiate them all specifically considering all
  optimizers of the network (set names, scopes... set optimization ops...).
  
  Arguments:
      loss: `Tensor`. Loss operation to evaluate network cost.
          Optimizer will use this cost function to train network.
      optimizer: `Optimizer`. Tensorflow Optimizer. The optimizer to
          use to train network.
      metric:  `Tensor`. The metric tensor to be used for evaluation.
      batch_size: `int`. Batch size for data feeded to this optimizer.
          Default: 64.
      ema: `float`. Exponential moving averages.
      trainable_vars: list of `tf.Variable`. List of trainable variables to
          use for training. Default: all trainable variables.
      shuffle: `bool`. Shuffle data.
      step_tensor: `tf.Tensor`. A variable holding training step. If not
          provided, it will be created. Early defining the step tensor
          might be useful for network creation, such as for learning rate
          decay.
      validation_monitors: `list` of `Tensor` objects.  List of variables
          to compute during validation, which are also used to produce
          summaries for output to TensorBoard.  For example, this can be
          used to periodically record a confusion matrix or AUC metric, 
          during training.  Each variable should have rank 1, i.e. 
          shape [None].
      validation_batch_size: `int` or None. If `int`, specifies the batch
          size to be used for the validation data feed; otherwise 
          defaults to being th esame as `batch_size`.
      name: `str`. A name for this class (optional).
      graph: `tf.Graph`. Tensorflow Graph to use for training. Default:
          default tf graph.
class_docstr_tok: ["TrainOp",".","TrainOp","represents","a","set","of","operation","used","for","optimizing","a","network",".","A","TrainOp","is","meant","to","hold","all","training","parameters","of","an","optimizer",".","`","Trainer","`","class","will","then","instantiate","them","all","specifically","considering","all","optimizers","of","the","network","(","set","names",",","scopes","...","set","optimization","ops","...",")",".","Arguments",":","loss",":","`","Tensor","`",".","Loss","operation","to","evaluate","network","cost",".","Optimizer","will","use","this","cost","function","to","train","network",".","optimizer",":","`","Optimizer","`",".","Tensorflow","Optimizer",".","The","optimizer","to","use","to","train","network",".","metric",":","`","Tensor","`",".","The","metric","tensor","to","be","used","for","evaluation",".","batch_size",":","`","int","`",".","Batch","size","for","data","feeded","to","this","optimizer",".","Default",":","64.","ema",":","`","float","`",".","Exponential","moving","averages",".","trainable_vars",":","list","of","`","tf.Variable","`",".","List","of","trainable","variables","to","use","for","training",".","Default",":","all","trainable","variables",".","shuffle",":","`","bool","`",".","Shuffle","data",".","step_tensor",":","`","tf.Tensor","`",".","A","variable","holding","training","step",".","If","not","provided",",","it","will","be","created",".","Early","defining","the","step","tensor","might","be","useful","for","network","creation",",","such","as","for","learning","rate","decay",".","validation_monitors",":","`","list","`","of","`","Tensor","`","objects",".","List","of","variables","to","compute","during","validation",",","which","are","also","used","to","produce","summaries","for","output","to","TensorBoard",".","For","example",",","this","can","be","used","to","periodically","record","a","confusion","matrix","or","AUC","metric",",","during","training",".","Each","variable","should","have","rank","1",",","i.e",".","shape","[","None","]",".","validation_batch_size",":","`","int","`","or","None",".","If","`","int","`",",","specifies","the","batch","size","to","be","used","for","the","validation","data","feed",";","otherwise","defaults","to","being","th","esame","as","`","batch_size","`",".","name",":","`","str","`",".","A","name","for","this","class","(","optional",")",".","graph",":","`","tf.Graph","`",".","Tensorflow","Graph","to","use","for","training",".","Default",":","default","tf","graph","."]
docstr: initialize_training_ops.
  
  Initialize all ops used for training. Because a network can have
  multiple optimizers, an id 'i' is allocated to differentiate them.
  This is meant to be used by `Trainer` when initializing all train ops.
  
  Arguments:
      i: `int`. This optimizer training process ID.
      session: `tf.Session`. The session used to train the network.
      tensorboard_verbose: `int`. Logs verbose. Supports:
          ```
          0 - Loss, Accuracy.
          1 - Loss, Accuracy, Gradients.
          2 - Loss, Accuracy, Gradients, Weights.
          3 - Loss, Accuracy, Gradients, Weights, Activations, Sparsity..
          ```
      clip_gradients: `float`. Option for clipping gradients.
code: def initialize_training_ops(
      self, i, session, tensorboard_verbose, clip_gradients
  ):
      """initialize_training_ops.
  
      Initialize all ops used for training. Because a network can have
      multiple optimizers, an id 'i' is allocated to differentiate them.
      This is meant to be used by `Trainer` when initializing all train ops.
  
      Arguments:
          i: `int`. This optimizer training process ID.
          session: `tf.Session`. The session used to train the network.
          tensorboard_verbose: `int`. Logs verbose. Supports:
              ```
              0 - Loss, Accuracy.
              1 - Loss, Accuracy, Gradients.
              2 - Loss, Accuracy, Gradients, Weights.
              3 - Loss, Accuracy, Gradients, Weights, Activations, Sparsity..
              ```
          clip_gradients: `float`. Option for clipping gradients.
      """
      self.session = session
  
      # Variables holding mean validation loss, accuracy, and validation
      # monitors, assigned after each model evaluation (by batch).
      # For visualization in Tensorboard.
      # Define variables, placeholders and assign ops.
      self.val_loss_T = tf.Variable(
          0.0, name="val_loss", trainable=False
      )
      self.val_acc_T = tf.Variable(0.0, name="val_acc", trainable=False)
      self.validation_monitors_T = [
          tf.Variable(
              0.0,
              name="%s_T" % v.name.rsplit(":", 1)[0],
              trainable=False,
          )
          for v in self.validation_monitors
      ]
  
      self.val_loss_P = tf.placeholder(
          dtype=tf.float32,
          name="placeholder/%s" % self.val_loss_T.name.rsplit(":")[0],
      )
      self.val_acc_P = tf.placeholder(
          dtype=tf.float32,
          name="placeholder/%s" % self.val_acc_T.name.rsplit(":")[0],
      )
      self.val_monitors_P = [
          tf.placeholder(
              dtype=tf.float32,
              name="placeholder/%s" % v.name.rsplit(":")[0],
          )
          for v in self.validation_monitors_T
      ]
  
      self.val_loss_assign = tf.assign(
          self.val_loss_T,
          self.val_loss_P,
          name="assign/%s" % self.val_loss_T.name.rsplit(":")[0],
      )
      self.val_acc_assign = tf.assign(
          self.val_acc_T,
          self.val_acc_P,
          name="assign/%s" % self.val_acc_T.name.rsplit(":")[0],
      )
      self.val_monitors_assign = [
          tf.assign(
              vmt, vmp, name="assign/%s" % vmt.name.rsplit(":")[0]
          )
          for vmt, vmp in zip(
              self.validation_monitors_T, self.val_monitors_P
          )
      ]
  
      # Creating the accuracy moving average, for better visualization.
      if self.metric is not None:
          self.acc_averages = tf.train.ExponentialMovingAverage(
              0.9, self.training_steps, name="moving_avg"
          )
          acc_avg_op = self.acc_averages.apply([self.metric])
      else:
          acc_avg_op = tf.no_op()
  
      # Compute total loss, which is the loss of all optimizers plus the
      # loss of all regularizers. Then, we summarize those losses for
      # visualization in Tensorboard.
      with tf.name_scope(self.name):
          lss = [self.loss] + tf.get_collection(
              tf.GraphKeys.REGULARIZATION_LOSSES
          )
          total_loss = tf.add_n(lss, name="Total_Loss")
          loss_avg_op = summaries.add_loss_summaries(
              total_loss,
              self.loss,
              regul_losses_collection_key=tf.GraphKeys.REGULARIZATION_LOSSES,
              name_prefix=self.scope_name,
              summaries_collection_key=self.name
              + "_training_summaries",
              exp_moving_avg=0.9,
              ema_num_updates=self.training_steps,
          )
  
          # Compute gradients operations
          with tf.control_dependencies([loss_avg_op, acc_avg_op]):
              self.grad = tf.gradients(total_loss, self.train_vars)
              if clip_gradients > 0.0:
                  self.grad, self.grad_norm = tf.clip_by_global_norm(
                      self.grad, clip_gradients
                  )
  
          self.grad = list(zip(self.grad, self.train_vars))
          self.apply_grad = self.optimizer.apply_gradients(
              grads_and_vars=self.grad,
              global_step=self.training_steps,
              name="apply_grad_op_" + str(i),
          )
  
          # Create other useful summary (weights, grads, activations...)
          # according to 'tensorboard_verbose' level.
          self.create_summaries(tensorboard_verbose)
  
          # Track the moving averages of trainable variables
          if self.ema > 0.0:
              var_averages = tf.train.ExponentialMovingAverage(
                  self.ema, self.training_steps
              )
              var_averages_op = var_averages.apply(self.train_vars)
  
              with tf.control_dependencies([var_averages_op]):
                  with tf.control_dependencies([self.apply_grad]):
                      self.train = tf.no_op(name="train_op_" + str(i))
          else:
              with tf.control_dependencies([self.apply_grad]):
                  self.train = tf.no_op(name="train_op_" + str(i))
  
imports: from __future__ import division, print_function, absolute_import
  import re
  import os
  import numpy
  import tensorflow.compat.v1
  from tensorflow.python.training import optimizer
  import tflearn
  from ..None import callbacks
  from ..config import init_training_mode
  from ..utils import to_list, id_generator, check_dir_name, standarize_dict, get_dict_first_element, make_batches, slice_array, check_scope_path, check_restore_tensor
  from ..None import data_flow
  from ..None import variables
  from ..None import utils
  from .summarizer import summaries, summarize, summarize_gradients, summarize_variables, summarize_activations
label_desc: Use image node_id 2 for calling the TrainOp obj's underlying member method code with example usage: obj.initialize_training_ops(i, session, tensorboard_verbose, clip_gradients) without return types