<p dir="auto">Ubuntu 14.04<br>
commit used: <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/tensorflow/tensorflow/commit/a154b6a7826760f30e368e84f51048218002d376/hovercard" href="https://github.com/tensorflow/tensorflow/commit/a154b6a7826760f30e368e84f51048218002d376"><tt>a154b6a</tt></a></p>
<p dir="auto">Simply changed num_gpus to 2. I have 2 GPUs. A 1060 and 970. Shows this error:</p>
<p dir="auto"><code class="notranslate">ValueError: Variable tower_1/tower_1/conv1/weight_loss/avg/biased does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope?</code></p>
<h3 dir="auto">Environment info</h3>
<p dir="auto">Operating System: Ubuntu</p>
<p dir="auto">Installed version of CUDA and cuDNN: 8.0 and 5</p>
<ol dir="auto">
<li>The commit hash (<code class="notranslate">git rev-parse HEAD</code>): <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/tensorflow/tensorflow/commit/3d41cf77d624aeee0482f92121a9300b29db2809/hovercard" href="https://github.com/tensorflow/tensorflow/commit/3d41cf77d624aeee0482f92121a9300b29db2809"><tt>3d41cf7</tt></a></li>
<li>The output of <code class="notranslate">bazel version</code>:<br>
Build label: 0.3.2<br>
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar<br>
Build time: Fri Oct 7 17:25:10 2016 (1475861110)<br>
Build timestamp: 1475861110<br>
Build timestamp as int: 1475861110</li>
</ol>
<h3 dir="auto">If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)</h3>
<p dir="auto">python cifar10_multi_gpu_train.py --num_gpus=2</p>
<p dir="auto">Both cifar10_train.py and cifar10_multi_gpu_train.py (without specifying num_gpus, so running on a single GPU) work.</p>
<h3 dir="auto">Logs or other output that would be helpful</h3>
<p dir="auto">I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally<br>
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally<br>
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally<br>
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally<br>
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally<br>
Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.<br>
Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.<br>
Traceback (most recent call last):<br>
File "cifar10_multi_gpu_train.py", line 280, in <br>
tf.app.run()<br>
File "/data/github/tensorflow/_python_build/tensorflow/python/platform/app.py", line 43, in run<br>
sys.exit(main(sys.argv[:1] + flags_passthrough))<br>
File "cifar10_multi_gpu_train.py", line 276, in main<br>
train()<br>
File "cifar10_multi_gpu_train.py", line 180, in train<br>
loss = tower_loss(scope)<br>
File "cifar10_multi_gpu_train.py", line 92, in tower_loss<br>
loss_averages_op = loss_averages.apply(losses + [total_loss])<br>
File "/data/github/tensorflow/_python_build/tensorflow/python/training/moving_averages.py", line 391, in apply<br>
self._averages[var], var, decay, zero_debias=zero_debias))<br>
File "/data/github/tensorflow/_python_build/tensorflow/python/training/moving_averages.py", line 70, in assign_moving_average<br>
update_delta = _zero_debias(variable, value, decay)<br>
File "/data/github/tensorflow/_python_build/tensorflow/python/training/moving_averages.py", line 177, in _zero_debias<br>
trainable=False)<br>
File "/data/github/tensorflow/_python_build/tensorflow/python/ops/variable_scope.py", line 1024, in get_variable<br>
custom_getter=custom_getter)<br>
File "/data/github/tensorflow/_python_build/tensorflow/python/ops/variable_scope.py", line 850, in get_variable<br>
custom_getter=custom_getter)<br>
File "/data/github/tensorflow/_python_build/tensorflow/python/ops/variable_scope.py", line 346, in get_variable<br>
validate_shape=validate_shape)<br>
File "/data/github/tensorflow/_python_build/tensorflow/python/ops/variable_scope.py", line 331, in _true_getter<br>
caching_device=caching_device, validate_shape=validate_shape)<br>
File "/data/github/tensorflow/_python_build/tensorflow/python/ops/variable_scope.py", line 650, in _get_single_variable<br>
"VarScope?" % name)<br>
ValueError: Variable tower_1/tower_1/conv1/weight_loss/avg/biased does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope?</p>