class_docstr: AllReduce Synchronizer.
  
  This AllReduce Synchronizer currently uses TensorFlow's `collective_device_ops`
  to insert their AllReduce ops into our graph.
  
  The class AllReduceSynchronizer class contains the following possible instantiations:
  
  1. spec=`auto`: single-node multiple devices, or cross-node AllReduce based on collective ops
  2. spec=`nccl`: single-node multiple devices, or cross-node AllReduce based on NCCL
  3. spec=`ring`/'tree', AllReduce with different reduction structures: ring, tree, etc.
  
  However note that it does not contain the following instantiations:
  
  1. shuffle reduce (reduce to CPU or GPU as in PS) + AllReduce across nodes
  2. any other types of hybrid reduction of PS and AllReduce.
class_docstr_tok: ["AllReduce","Synchronizer",".","This","AllReduce","Synchronizer","currently","uses","TensorFlow","'s","`","collective_device_ops","`","to","insert","their","AllReduce","ops","into","our","graph",".","The","class","AllReduceSynchronizer","class","contains","the","following","possible","instantiations",":","1.","spec=","`","auto","`",":","single-node","multiple","devices",",","or","cross-node","AllReduce","based","on","collective","ops","2.","spec=","`","nccl","`",":","single-node","multiple","devices",",","or","cross-node","AllReduce","based","on","NCCL","3.","spec=","`","ring","`","\/'tree","'",",","AllReduce","with","different","reduction","structures",":","ring",",","tree",",","etc",".","However","note","that","it","does","not","contain","the","following","instantiations",":","1.","shuffle","reduce","(","reduce","to","CPU","or","GPU","as","in","PS",")","+","AllReduce","across","nodes","2.","any","other","types","of","hybrid","reduction","of","PS","and","AllReduce","."]
code: def __init__(self, config: synchronizers_pb2.AllReduceSynchronizer):
      self._spec = synchronizers_pb2.AllReduceSynchronizer.Spec.Name(
          config.spec
      )
      if (
          autodist.float_major_minor_tf_version < 1.15
          or autodist.float_major_minor_tf_version < 2.1
      ):
          logging.warning(
              'Collective synchronizer spec "{}" a.k.a communication_hint has no effect '
              "until tensorflow-gpu 1.x>= 1.15 or 2.x>=2.1. It may cause error currently.".format(
                  self._spec
              )
          )
          self._spec = None
  
      self._compressor_type = (
          synchronizers_pb2.AllReduceSynchronizer.Compressor.Name(
              config.compressor
          )
      )
  
      # Collective ops within the same group will be merged by the scoped optimizer.
      # Normally the group index shall be smaller than the number of variables in the graph; this kernel assumes
      # the strategy will validate the group assignments are legitimate.
      self._group = config.group
      super().__init__()
  
imports: from collections import defaultdict
  from tensorflow.python import ops
  from tensorflow.python.framework import device_spec
  from tensorflow.python.ops import collective_ops
  import autodist
  from autodist.const import ENV
  from autodist.kernel.common.utils import get_consumers, update_consumers, replica_prefix, get_control_consumers, update_control_consumers
  from autodist.kernel.common.utils import get_op_name
  from autodist.kernel.synchronization.collective_key import get_collective_keys
  from autodist.kernel.synchronization.compressor import Compressor, CollectiveOpsConfig
  from autodist.kernel.synchronization.synchronizer import Synchronizer
  from autodist.proto import synchronizers_pb2
  from autodist.utils import logging
label_desc: Use image node_id 1 to create a new AllReduceSynchronizer object from inherited base classes: Synchronizer with example: obj = AllReduceSynchronizer(config)