class_docstr: Class to synchronize, aggregate gradients and pass them to the optimizer.
  
  This class is deprecated. For synchronous training, please use [Distribution
  Strategies](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/distribute).
  
  In a typical asynchronous training environment, it's common to have some
  stale gradients. For example, with a N-replica asynchronous training,
  gradients will be applied to the variables N times independently. Depending
  on each replica's training speed, some gradients might be calculated from
  copies of the variable from several steps back (N-1 steps on average). This
  optimizer avoids stale gradients by collecting gradients from all replicas,
  averaging them, then applying them to the variables in one shot, after
  which replicas can fetch the new variables and continue.
  
  The following accumulators/queue are created:
  
  * N `gradient accumulators`, one per variable to train. Gradients are pushed
    to them and the chief worker will wait until enough gradients are collected
    and then average them before applying to variables. The accumulator will
    drop all stale gradients (more details in the accumulator op).
  * 1 `token` queue where the optimizer pushes the new global_step value after
    all variables are updated.
  
  The following local variable is created:
  * `sync_rep_local_step`, one per replica. Compared against the global_step in
    each accumulator to check for staleness of the gradients.
  
  The optimizer adds nodes to the graph to collect gradients and pause the
  trainers until variables are updated.
  For the Parameter Server job:
  
  1. An accumulator is created for each variable, and each replica pushes the
     gradients into the accumulators instead of directly applying them to the
     variables.
  2. Each accumulator averages once enough gradients (replicas_to_aggregate)
     have been accumulated.
  3. Apply the averaged gradients to the variables.
  4. Only after all variables have been updated, increment the global step.
  5. Only after step 4, pushes `global_step` in the `token_queue`, once for
     each worker replica. The workers can now fetch the global step, use it to
     update its local_step variable and start the next batch. Please note that
     some workers can consume multiple minibatches, while some may not consume
     even one. This is because each worker fetches minibatches as long as
     a token exists. If one worker is stuck for some reason and does not
     consume a token, another worker can use it.
  
  For the replicas:
  
  1. Start a step: fetch variables and compute gradients.
  2. Once the gradients have been computed, push them into gradient
     accumulators. Each accumulator will check the staleness and drop the stale.
  3. After pushing all the gradients, dequeue an updated value of global_step
     from the token queue and record that step to its local_step variable. Note
     that this is effectively a barrier.
  4. Start the next batch.
  
  ### Usage
  
  ```python
  # Create any optimizer to update the variables, say a simple SGD:
  opt = GradientDescentOptimizer(learning_rate=0.1)
  
  # Wrap the optimizer with sync_replicas_optimizer with 50 replicas: at each
  # step the optimizer collects 50 gradients before applying to variables.
  # Note that if you want to have 2 backup replicas, you can change
  # total_num_replicas=52 and make sure this number matches how many physical
  # replicas you started in your job.
  opt = tf.compat.v1.train.SyncReplicasOptimizer(opt, replicas_to_aggregate=50,
                                 total_num_replicas=50)
  
  # Some models have startup_delays to help stabilize the model but when using
  # sync_replicas training, set it to 0.
  
  # Now you can call `minimize()` or `compute_gradients()` and
  # `apply_gradients()` normally
  training_op = opt.minimize(total_loss, global_step=self.global_step)
  
  
  # You can create the hook which handles initialization and queues.
  sync_replicas_hook = opt.make_session_run_hook(is_chief)
  ```
  
  In the training program, every worker will run the train_op as if not
  synchronized.
  
  ```python
  with training.MonitoredTrainingSession(
      master=workers[worker_id].target, is_chief=is_chief,
      hooks=[sync_replicas_hook]) as mon_sess:
    while not mon_sess.should_stop():
      mon_sess.run(training_op)
  ```
  
  To use SyncReplicasOptimizer with an `Estimator`, you need to send
  sync_replicas_hook while calling the fit.
  ```python
  my_estimator = DNNClassifier(..., optimizer=opt)
  my_estimator.fit(..., hooks=[sync_replicas_hook])
  ```
class_docstr_tok: ["Class","to","synchronize",",","aggregate","gradients","and","pass","them","to","the","optimizer",".","This","class","is","deprecated",".","For","synchronous","training",",","please","use","[","Distribution","Strategies","]","(","https",":","\/\/github.com\/tensorflow\/tensorflow\/tree\/master\/tensorflow\/contrib\/distribute",")",".","In","a","typical","asynchronous","training","environment",",","it","'s","common","to","have","some","stale","gradients",".","For","example",",","with","a","N-replica","asynchronous","training",",","gradients","will","be","applied","to","the","variables","N","times","independently",".","Depending","on","each","replica","'s","training","speed",",","some","gradients","might","be","calculated","from","copies","of","the","variable","from","several","steps","back","(","N-1","steps","on","average",")",".","This","optimizer","avoids","stale","gradients","by","collecting","gradients","from","all","replicas",",","averaging","them",",","then","applying","them","to","the","variables","in","one","shot",",","after","which","replicas","can","fetch","the","new","variables","and","continue",".","The","following","accumulators\/queue","are","created",":","*","N","`","gradient","accumulators","`",",","one","per","variable","to","train",".","Gradients","are","pushed","to","them","and","the","chief","worker","will","wait","until","enough","gradients","are","collected","and","then","average","them","before","applying","to","variables",".","The","accumulator","will","drop","all","stale","gradients","(","more","details","in","the","accumulator","op",")",".","*","1","`","token","`","queue","where","the","optimizer","pushes","the","new","global_step","value","after","all","variables","are","updated",".","The","following","local","variable","is","created",":","*","`","sync_rep_local_step","`",",","one","per","replica",".","Compared","against","the","global_step","in","each","accumulator","to","check","for","staleness","of","the","gradients",".","The","optimizer","adds","nodes","to","the","graph","to","collect","gradients","and","pause","the","trainers","until","variables","are","updated",".","For","the","Parameter","Server","job",":","1",".","An","accumulator","is","created","for","each","variable",",","and","each","replica","pushes","the","gradients","into","the","accumulators","instead","of","directly","applying","them","to","the","variables",".","2",".","Each","accumulator","averages","once","enough","gradients","(","replicas_to_aggregate",")","have","been","accumulated",".","3",".","Apply","the","averaged","gradients","to","the","variables",".","4",".","Only","after","all","variables","have","been","updated",",","increment","the","global","step",".","5",".","Only","after","step","4",",","pushes","`","global_step","`","in","the","`","token_queue","`",",","once","for","each","worker","replica",".","The","workers","can","now","fetch","the","global","step",",","use","it","to","update","its","local_step","variable","and","start","the","next","batch",".","Please","note","that","some","workers","can","consume","multiple","minibatches",",","while","some","may","not","consume","even","one",".","This","is","because","each","worker","fetches","minibatches","as","long","as","a","token","exists",".","If","one","worker","is","stuck","for","some","reason","and","does","not","consume","a","token",",","another","worker","can","use","it",".","For","the","replicas",":","1",".","Start","a","step",":","fetch","variables","and","compute","gradients",".","2",".","Once","the","gradients","have","been","computed",",","push","them","into","gradient","accumulators",".","Each","accumulator","will","check","the","staleness","and","drop","the","stale",".","3",".","After","pushing","all","the","gradients",",","dequeue","an","updated","value","of","global_step","from","the","token","queue","and","record","that","step","to","its","local_step","variable",".","Note","that","this","is","effectively","a","barrier",".","4",".","Start","the","next","batch",".","#","#","#","Usage","``","`","python","#","Create","any","optimizer","to","update","the","variables",",","say","a","simple","SGD",":","opt","=","GradientDescentOptimizer","(","learning_rate=0.1",")","#","Wrap","the","optimizer","with","sync_replicas_optimizer","with","50","replicas",":","at","each","#","step","the","optimizer","collects","50","gradients","before","applying","to","variables",".","#","Note","that","if","you","want","to","have","2","backup","replicas",",","you","can","change","#","total_num_replicas=52","and","make","sure","this","number","matches","how","many","physical","#","replicas","you","started","in","your","job",".","opt","=","tf.compat.v1.train.SyncReplicasOptimizer","(","opt",",","replicas_to_aggregate=50",",","total_num_replicas=50",")","#","Some","models","have","startup_delays","to","help","stabilize","the","model","but","when","using","#","sync_replicas","training",",","set","it","to","0",".","#","Now","you","can","call","`","minimize","(",")","`","or","`","compute_gradients","(",")","`","and","#","`","apply_gradients","(",")","`","normally","training_op","=","opt.minimize","(","total_loss",",","global_step=self.global_step",")","#","You","can","create","the","hook","which","handles","initialization","and","queues",".","sync_replicas_hook","=","opt.make_session_run_hook","(","is_chief",")","``","`","In","the","training","program",",","every","worker","will","run","the","train_op","as","if","not","synchronized",".","``","`","python","with","training.MonitoredTrainingSession","(","master=workers","[","worker_id","]",".target",",","is_chief=is_chief",",","hooks=","[","sync_replicas_hook","]",")","as","mon_sess",":","while","not","mon_sess.should_stop","(",")",":","mon_sess.run","(","training_op",")","``","`","To","use","SyncReplicasOptimizer","with","an","`","Estimator","`",",","you","need","to","send","sync_replicas_hook","while","calling","the","fit",".","``","`","python","my_estimator","=","DNNClassifier","(","...",",","optimizer=opt",")","my_estimator.fit","(","...",",","hooks=","[","sync_replicas_hook","]",")","``","`"]
docstr: Creates a hook to handle SyncReplicasHook ops such as initialization.
docstr_tok: ["Creates","a","hook","to","handle","SyncReplicasHook","ops","such","as","initialization","."]
code: def make_session_run_hook(self, is_chief, num_tokens=-1):
      """Creates a hook to handle SyncReplicasHook ops such as initialization."""
      return _SyncReplicasOptimizerHook(self, is_chief, num_tokens)
  
imports: from tensorflow.python.distribute import distribute_lib
  from tensorflow.python.framework import indexed_slices
  from tensorflow.python.framework import ops
  from tensorflow.python.framework import tensor
  from tensorflow.python.ops import array_ops
  from tensorflow.python.ops import control_flow_ops
  from tensorflow.python.ops import data_flow_ops
  from tensorflow.python.ops import state_ops
  from tensorflow.python.ops import variable_v1
  from tensorflow.python.ops import variables
  from tensorflow.python.platform import tf_logging
  from tensorflow.python.training import optimizer
  from tensorflow.python.training import queue_runner
  from tensorflow.python.training import session_manager
  from tensorflow.python.training import session_run_hook
  from tensorflow.python.util import deprecation
  from tensorflow.python.util.tf_export import tf_export
label_desc: Use image node_id 9 for calling the SyncReplicasOptimizer obj's underlying member method code with example usage: obj.make_session_run_hook(is_chief, num_tokens) and returns: _SyncReplicasOptimizerHook