class_docstr: AllReduce Synchronizer.
  
  This AllReduce Synchronizer currently uses TensorFlow's `collective_device_ops`
  to insert their AllReduce ops into our graph.
  
  The class AllReduceSynchronizer class contains the following possible instantiations:
  
  1. spec=`auto`: single-node multiple devices, or cross-node AllReduce based on collective ops
  2. spec=`nccl`: single-node multiple devices, or cross-node AllReduce based on NCCL
  3. spec=`ring`/'tree', AllReduce with different reduction structures: ring, tree, etc.
  
  However note that it does not contain the following instantiations:
  
  1. shuffle reduce (reduce to CPU or GPU as in PS) + AllReduce across nodes
  2. any other types of hybrid reduction of PS and AllReduce.
class_docstr_tok: ["AllReduce","Synchronizer",".","This","AllReduce","Synchronizer","currently","uses","TensorFlow","'s","`","collective_device_ops","`","to","insert","their","AllReduce","ops","into","our","graph",".","The","class","AllReduceSynchronizer","class","contains","the","following","possible","instantiations",":","1.","spec=","`","auto","`",":","single-node","multiple","devices",",","or","cross-node","AllReduce","based","on","collective","ops","2.","spec=","`","nccl","`",":","single-node","multiple","devices",",","or","cross-node","AllReduce","based","on","NCCL","3.","spec=","`","ring","`","\/'tree","'",",","AllReduce","with","different","reduction","structures",":","ring",",","tree",",","etc",".","However","note","that","it","does","not","contain","the","following","instantiations",":","1.","shuffle","reduce","(","reduce","to","CPU","or","GPU","as","in","PS",")","+","AllReduce","across","nodes","2.","any","other","types","of","hybrid","reduction","of","PS","and","AllReduce","."]
docstr: Share the initializers of all replica variables to use initializer on replica=master_replica.
docstr_tok: ["Share","the","initializers","of","all","replica","variables","to","use","initializer","on","replica=master_replica","."]
code: def _share_initializer(
      self, graph_item, var_op_name, master_replica=0
  ):
      """Share the initializers of all replica variables to use initializer on replica=master_replica."""
      # find the initial value of the var on master_replica
      master_var_op = graph_item.graph.get_operation_by_name(
          ops.prepend_name_scope(
              var_op_name, replica_prefix(master_replica)
          )
      )
      master_var = graph_item.trainable_var_op_to_var[master_var_op]
      master_init_tensor = graph_item.graph.get_tensor_by_name(
          master_var.initial_value.name
      )
      master_init_op = master_init_tensor.op
      # set the device of the init ops to reside on the chief device
      master_init_device = device_spec.DeviceSpecV2.from_string(
          master_init_op.device
      ).replace(task=0)
      master_init_op._set_device_from_string(
          master_init_device.to_string()
      )
  
      for i in range(0, self.num_replicas):
          if i == master_replica:
              continue
          var_op = graph_item.graph.get_operation_by_name(
              ops.prepend_name_scope(var_op_name, replica_prefix(i))
          )
          var = graph_item.trainable_var_op_to_var[var_op]
          init_op = graph_item.graph.get_tensor_by_name(
              var.initial_value.name
          ).op
          init_assign_op = get_consumers(init_op)[0]
          init_assign_op._update_input(1, master_init_tensor)
  
imports: from collections import defaultdict
  from tensorflow.python import ops
  from tensorflow.python.framework import device_spec
  from tensorflow.python.ops import collective_ops
  import autodist
  from autodist.const import ENV
  from autodist.kernel.common.utils import get_consumers, update_consumers, replica_prefix, get_control_consumers, update_control_consumers
  from autodist.kernel.common.utils import get_op_name
  from autodist.kernel.synchronization.collective_key import get_collective_keys
  from autodist.kernel.synchronization.compressor import Compressor, CollectiveOpsConfig
  from autodist.kernel.synchronization.synchronizer import Synchronizer
  from autodist.proto import synchronizers_pb2
  from autodist.utils import logging
label_desc: Use image node_id 5 for calling the AllReduceSynchronizer obj's underlying member method code with example usage: obj._share_initializer(graph_item, var_op_name, master_replica) without return types