class_docstr: Data-parallel training in "replicated" mode,
  where each GPU contains a replicate of the whole model.
  It will build one tower on each GPU under its own variable scope.
  Each gradient update is averaged or summed across or GPUs through NCCL.
  
  It is an equivalent of ``--variable_update=replicated`` in
  `tensorflow/benchmarks <https://github.com/tensorflow/benchmarks>`_.
class_docstr_tok: ["Data-parallel","training","in","``","replicated","''","mode",",","where","each","GPU","contains","a","replicate","of","the","whole","model",".","It","will","build","one","tower","on","each","GPU","under","its","own","variable","scope",".","Each","gradient","update","is","averaged","or","summed","across","or","GPUs","through","NCCL",".","It","is","an","equivalent","of","``","--","variable_update=replicated","``","in","`","tensorflow\/benchmarks","<","https",":","\/\/github.com\/tensorflow\/benchmarks",">","`","_","."]
code: def __init__(self, towers, average, mode):
      super(SyncMultiGPUReplicatedBuilder, self).__init__(towers)
      self._average = average
      assert mode in [
          "nccl",
          "cpu",
          "hierarchical",
          "gpu",
          "collective",
      ], mode
      self._mode = mode
  
      if self._mode == "hierarchical" and len(towers) != 8:
          raise ValueError("mode='hierarchical' require 8 GPUs.")
  
imports: import copy
  import pprint
  import re
  from abc import ABCMeta, abstractmethod
  from contextlib import contextmanager
  import six
  import tensorflow
  from ..compat import tfv1
  from ..tfutils.common import get_tf_version_tuple
  from ..tfutils.gradproc import ScaleGradient
  from ..tfutils.tower import TrainTowerContext
  from ..utils import logger
  from ..utils.develop import HIDE_DOC
  from .utils import GradientPacker, LeastLoadedDeviceSetter, aggregate_grads_colocate, allreduce_grads_naive, allreduce_grads, allreduce_grads_hierarchical, merge_grad_list, override_to_local_variable, split_grad_list
label_desc: Use image node_id 1 to create a new SyncMultiGPUReplicatedBuilder object from inherited base classes: DataParallelBuilder with example: obj = SyncMultiGPUReplicatedBuilder(towers, average, mode)