title: Distributed Tensorflow Independent Weights
body: <p>I'm new to distributed tensorflow and I'm trying to implement an asynchronous algorithm where each worker has its own weights, but can access the weights of other workers globally. The intent is that during each training step, the worker has an option of either to continue training its current weights, or inherit the weights of another worker.</p>
  
  <p>I've scoured many examples on the internet regarding data parallelism where each device has the same model / graph, but it seems like in all their cases the weights are shared, which is not what I want.</p>
  
  <p>So my question is, how can I setup the same graph on each device, but keep the trainable weights independent? And how can I create a global variable where all workers can effectively dump or retrieve weights into it? I'm assuming this will have to be set up on the parameter server. </p>
  
  <p>Thanks. </p>