class_docstr: Synchronizer.
  
  Given a variable, can modify the TF Graph to synchronize its
  gradients in either an in-graph or a between-graph fashion.
  
  - In-graph means the synchronization happens in one `tf.Graph`
  - Between-graph means the synchronization happens across
      multiple `tf.Graphs` (e.g., each worker has its own graph)
code: def __init__(self):
      self.num_workers = None
      self.num_replicas = None
      self.worker_device = None
      self.worker_id = None
      self.var_op_to_agg_grad = None
      self.var_op_to_accum_apply_op = None
      self.is_chief = None
      self.all_canonical_replica_devices = None
  
file_path: autodist/autodist/kernel/synchronization/synchronizer.py
imports: from abc import ABC, abstractmethod
  from tensorflow.python import ops
  from autodist.kernel.common.utils import get_op_name, update_consumers, update_control_consumers, replica_prefix, strip_replica_prefix, get_index_from_tensor_name
label_desc: Use image node_id 1 to create a new Synchronizer object from inherited base classes: ABC with example: obj = Synchronizer()