class_docstr: Replicates input, parameters and compute over multiple accelerators.
  
  ``Replicator`` is a TensorFlow "Distribution Strategy" implementing the
  programming model described in the TF-Replicator paper
  :cite:`buchlovsky2019tf` and TensorFlow RFC
  :cite:`buchlovsky2019distribution`. ``Replicator`` enables data-parallel
  training across multiple accelerators on a single machine, it supports
  eager execution and :tf:`function`.
  
  To get started create a ``Replicator`` instance:
  
      >>> replicator = snt.distribute.Replicator()
  
  Replicator provides a scope inside which any new :tf:`Variable`\ s will be
  replicated across all local devices:
  
      >>> with replicator.scope():
      ...    mod = snt.Linear(32)
  
  Additionally replicator provides utility functions to apply a module in
  parallel on multiple devices. First we need to define some computation that
  runs on each GPU. The "replica context" object provides us a way to
  communicate between replicas (e.g. to perform an ``all_reduce``):
  
      >>> def forward():
      ...   # Compute a random output on each GPU.
      ...   x = tf.random.normal([8, 28 * 28])
      ...   y = mod(x)
      ...   # Synchronize the value of `y` between all GPUs.
      ...   ctx = tf.distribute.get_replica_context()
      ...   y = ctx.all_reduce("mean", y)
      ...   return y
  
  Finally we use the run API to apply ``forward`` in parallel on all accelerator
  devices:
  
      >>> per_replica_y = replicator.run(forward)
class_docstr_tok: ["Replicates","input",",","parameters","and","compute","over","multiple","accelerators",".","``","Replicator","``","is","a","TensorFlow","``","Distribution","Strategy","''","implementing","the","programming","model","described","in","the","TF-Replicator","paper",":","cite",":","`","buchlovsky2019tf","`","and","TensorFlow","RFC",":","cite",":","`","buchlovsky2019distribution","`",".","``","Replicator","``","enables","data-parallel","training","across","multiple","accelerators","on","a","single","machine",",","it","supports","eager","execution","and",":","tf",":","`","function","`",".","To","get","started","create","a","``","Replicator","``","instance",":",">",">",">","replicator","=","snt.distribute.Replicator","(",")","Replicator","provides","a","scope","inside","which","any","new",":","tf",":","`","Variable","`","\\","s","will","be","replicated","across","all","local","devices",":",">",">",">","with","replicator.scope","(",")",":","...","mod","=","snt.Linear","(","32",")","Additionally","replicator","provides","utility","functions","to","apply","a","module","in","parallel","on","multiple","devices",".","First","we","need","to","define","some","computation","that","runs","on","each","GPU",".","The","``","replica","context","''","object","provides","us","a","way","to","communicate","between","replicas","(","e.g",".","to","perform","an","``","all_reduce","``",")",":",">",">",">","def","forward","(",")",":","...","#","Compute","a","random","output","on","each","GPU",".","...","x","=","tf.random.normal","(","[","8",",","28","*","28","]",")","...","y","=","mod","(","x",")","...","#","Synchronize","the","value","of","`","y","`","between","all","GPUs",".","...","ctx","=","tf.distribute.get_replica_context","(",")","...","y","=","ctx.all_reduce","(","``","mean","''",",","y",")","...","return","y","Finally","we","use","the","run","API","to","apply","``","forward","``","in","parallel","on","all","accelerator","devices",":",">",">",">","per_replica_y","=","replicator.run","(","forward",")"]
code: def scope(self):
      with contextlib.ExitStack() as stack:
          stack.enter_context(super().scope())
          stack.enter_context(
              tf.variable_creator_scope(replica_local_creator)
          )
          yield
  
file_path: sonnet/sonnet/src/distribute/replicator.py
imports: from typing import Callable, TypeVar
  from absl import logging
  import contextlib
  from sonnet.src import initializers
  import tensorflow
label_desc: Use image node_id 1 for calling the Replicator obj's underlying member method code with example usage: obj.scope() without return types