<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="280536217" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/15216" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/15216/hovercard" href="https://github.com/tensorflow/tensorflow/issues/15216">#15216</a><br>
i have a issue, but nobody help me to solve it</p>
<p dir="auto">i use tensorflow to distributed trainning models, i use the partition valriables to store an array data, when the data is not so bigger, everything looks ok,but when the array data become larger, when the session initialize, the partition variables can not initialized and the session will wait util time out.</p>
<p dir="auto">Describe the problem</p>
<p dir="auto">Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request. the feat_info can initialize successfully, but the adj_info cannot initialized. the adj_info is larger than feat_info</p>
<p dir="auto">i use monitored_session or supervisor to do the initialize_variables, when data is not so large, it will initialize successfully, my machine have more than 300GB memory is enough for the larger variable</p>
<p dir="auto">Have I written custom code<br>
yes<br>
OS Platform and Distribution<br>
linux platform<br>
TensorFlow installed from<br>
N/A<br>
TensorFlow version<br>
r1.4<br>
Bazel version<br>
CUDA/cuDNN version<br>
no gpu<br>
GPU model and memory<br>
no gpu<br>
Exact command to reproduce<br>
N/A</p>
<p dir="auto">Source code / logs</p>
<p dir="auto">i use ps_num = 4, worker_num =4 and i also try some other distributed config, like ps_num=1, worker_num=4, the result is the same<br>
source code:<br>
with tf.device(tf.train.replica_device_setter(<br>
worker_device="/job:worker/task:%d" % task_id,<br>
cluster=cluster_spec)):</p>
<p dir="auto">feat_info = tf.get_variable("feature_info", (len(id_map),FLAGS.features_column), tf.float32, trainable=False, partitioner=tf.fixed_size_partitioner(num_workers))<br>
adj_info = tf.get_variable("adj_info", (len(id_map),FLAGS.max_degree), tf.int64, trainable=False, partitioner=tf.fixed_size_partitioner(num_workers))</p>
<p dir="auto">with tf.device('/job:worker/task:%d' %task_id):<br>
adj_local = tf.Variable(tf.constant(minibatch.adj, dtype=tf.int64), trainable=False, name="adj_local", collections=[tf.GraphKeys.LOCAL_VARIABLES])<br>
feat_local = tf.Variable(tf.constant(features, dtype=tf.float32), trainable=False, name="feat_local", collections=[tf.GraphKeys.LOCAL_VARIABLES])</p>
<p dir="auto">length, begin, end = split_node_by_task(len(id_map), task_id, num_workers)<br>
adj = tf.nn.embedding_lookup(adj_info, [x for x in range(begin, end)])<br>
adj = adj_local</p>
<p dir="auto">feat = tf.nn.embedding_lookup(feat_info, [x for x in range(begin, end)])<br>
feat = feat_local<br>
log:<br>
2017-12-08 23:54:17.377290: I tensorflow/core/distributed_runtime/master_session.cc:998] Start master session c2b3ba9b700261ba with config:<br>
INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: adj_info/part_0, adj_info/part_1, adj_info/part_2, adj_info/part_3, adj_info/part_4, adj_info/part_5, adj_info/part_6, adj_info/part_7, adj_info/part_8, adj_info/part_9, adj_info/part_10, adj_info/part_11, adj_info/part_12, adj_info/part_13, adj_info/part_14, adj_info/part_15<br>
2017-12-09 00:00:35.637019: I tensorflow/core/distributed_runtime/master_session.cc:998] Start master session f35fcf332e3908ec with config:<br>
INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: adj_info/part_0, adj_info/part_1, adj_info/part_2, adj_info/part_3, adj_info/part_4, adj_info/part_5, adj_info/part_6, adj_info/part_7, adj_info/part_8, adj_info/part_9, adj_info/part_10, adj_info/part_11, adj_info/part_12, adj_info/part_13, adj_info/part_14, adj_info/part_15<br>
and it will alway waiting adj_info to initialize</p>