<p dir="auto">While trying to optimize training of a network by moving most work to the GPU, I encountered the following error below.  Although this references a variable defined in the batch_normalization helper class I've used, removing that class (replacing all calls to it with an identify function) results in a similar error elsewhere.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;train_resnet.py&quot;, line 106, in &lt;module&gt;
    tf.app.run()
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/platform/app.py&quot;, line 30, in run
    sys.exit(main(sys.argv))
  File &quot;train_resnet.py&quot;, line 103, in main
    run_training()
  File &quot;train_resnet.py&quot;, line 97, in run_training
    sess.run(tf.initialize_all_variables())
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/client/session.py&quot;, line 333, in run
    run_metadata_ptr)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/client/session.py&quot;, line 573, in _run
    feed_dict_string, options, run_metadata)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/client/session.py&quot;, line 648, in _do_run
    target_list, options, run_metadata)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/client/session.py&quot;, line 668, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'gradients/resnet_module_1/bn/moments/moments/mean_ss_grad/Maximum/y': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available
         [[Node: gradients/resnet_module_1/bn/moments/moments/mean_ss_grad/Maximum/y = Const[_class=[&quot;loc:@resnet_module_1/bn/moments/moments/mean_ss&quot;], dtype=DT_INT32, value=Tensor&lt;type: int32 shape: [] values: 1&gt;, _device=&quot;/device:GPU:0&quot;]()]]
Caused by op 'gradients/resnet_module_1/bn/moments/moments/mean_ss_grad/Maximum/y', defined at:
  File &quot;train_resnet.py&quot;, line 106, in &lt;module&gt;
    tf.app.run()
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/platform/app.py&quot;, line 30, in run
    sys.exit(main(sys.argv))
  File &quot;train_resnet.py&quot;, line 103, in main
    run_training()
  File &quot;train_resnet.py&quot;, line 89, in run_training
    train_op = opt.minimize(loss, colocate_gradients_with_ops=True, aggregation_method=2)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/training/optimizer.py&quot;, line 193, in minimize
    grad_loss=grad_loss)
  File &quot;/data/Ray/policy_network/gpu/clipopt.py&quot;, line 9, in compute_gradients
    *args, **kwargs)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/training/optimizer.py&quot;, line 250, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/gradients.py&quot;, line 481, in gradients
    in_grads = _AsList(grad_fn(op, *out_grads))
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/math_grad.py&quot;, line 41, in _SumGrad
    tile_scaling = _safe_shape_div(input_shape, output_shape_kept_dims)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/math_grad.py&quot;, line 33, in _safe_shape_div
    return x // math_ops.maximum(y, 1)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py&quot;, line 1173, in maximum
    result = _op_def_lib.apply_op(&quot;Maximum&quot;, x=x, y=y, name=name)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/op_def_library.py&quot;, line 455, in apply_op
    as_ref=input_arg.is_ref)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py&quot;, line 620, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/constant_op.py&quot;, line 179, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/constant_op.py&quot;, line 166, in constant
    attrs={&quot;value&quot;: tensor_value, &quot;dtype&quot;: dtype_value}, name=name).outputs[0]
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py&quot;, line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py&quot;, line 1224, in __init__
    self._traceback = _extract_stack()

...which was originally created as op 'resnet_module_1/bn/moments/moments/mean_ss', defined at:
  File &quot;train_resnet.py&quot;, line 106, in &lt;module&gt;
    tf.app.run()
[elided 1 identical lines from previous traceback]
  File &quot;train_resnet.py&quot;, line 103, in main
    run_training()
  File &quot;train_resnet.py&quot;, line 77, in run_training
    depth=FLAGS.num_features)
  File &quot;train_resnet.py&quot;, line 45, in model
    model = res_3x3_pair(model, depth, &quot;resnet_module_{}&quot;.format(idx + 1), training_switch, keep_prob_var)
  File &quot;train_resnet.py&quot;, line 29, in res_3x3_pair
    normed = tf.nn.relu(batch_norm(data, depth, training_switch))
  File &quot;/data/Ray/policy_network/gpu/batchnorm.py&quot;, line 22, in batch_norm
    batch_mean, batch_var = tf.nn.moments(x, [0, 1, 2], name='moments')
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/nn.py&quot;, line 712, in moments
    name=name)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/nn.py&quot;, line 649, in sufficient_statistics
    m_ss = math_ops.reduce_sum(m_ss, axes, keep_dims=keep_dims, name=&quot;mean_ss&quot;)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py&quot;, line 909, in reduce_sum
    keep_dims, name=name)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py&quot;, line 2087, in _sum
    keep_dims=keep_dims, name=name)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/op_def_library.py&quot;, line 704, in apply_op
    op_def=op_def)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py&quot;, line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File &quot;/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py&quot;, line 1224, in __init__
    self._traceback = _extract_stack()"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "train_resnet.py", line 106, in &lt;module&gt;
    tf.app.run()
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "train_resnet.py", line 103, in main
    run_training()
  File "train_resnet.py", line 97, in run_training
    sess.run(tf.initialize_all_variables())
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 333, in run
    run_metadata_ptr)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 573, in _run
    feed_dict_string, options, run_metadata)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 648, in _do_run
    target_list, options, run_metadata)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 668, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'gradients/resnet_module_1/bn/moments/moments/mean_ss_grad/Maximum/y': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available
         [[Node: gradients/resnet_module_1/bn/moments/moments/mean_ss_grad/Maximum/y = Const[_class=["loc:@resnet_module_1/bn/moments/moments/mean_ss"], dtype=DT_INT32, value=Tensor&lt;type: int32 shape: [] values: 1&gt;, _device="/device:GPU:0"]()]]
Caused by op 'gradients/resnet_module_1/bn/moments/moments/mean_ss_grad/Maximum/y', defined at:
  File "train_resnet.py", line 106, in &lt;module&gt;
    tf.app.run()
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "train_resnet.py", line 103, in main
    run_training()
  File "train_resnet.py", line 89, in run_training
    train_op = opt.minimize(loss, colocate_gradients_with_ops=True, aggregation_method=2)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/training/optimizer.py", line 193, in minimize
    grad_loss=grad_loss)
  File "/data/Ray/policy_network/gpu/clipopt.py", line 9, in compute_gradients
    *args, **kwargs)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/training/optimizer.py", line 250, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/gradients.py", line 481, in gradients
    in_grads = _AsList(grad_fn(op, *out_grads))
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/math_grad.py", line 41, in _SumGrad
    tile_scaling = _safe_shape_div(input_shape, output_shape_kept_dims)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/math_grad.py", line 33, in _safe_shape_div
    return x // math_ops.maximum(y, 1)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 1173, in maximum
    result = _op_def_lib.apply_op("Maximum", x=x, y=y, name=name)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/op_def_library.py", line 455, in apply_op
    as_ref=input_arg.is_ref)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 620, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/constant_op.py", line 179, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/constant_op.py", line 166, in constant
    attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1224, in __init__
    self._traceback = _extract_stack()

...which was originally created as op 'resnet_module_1/bn/moments/moments/mean_ss', defined at:
  File "train_resnet.py", line 106, in &lt;module&gt;
    tf.app.run()
[elided 1 identical lines from previous traceback]
  File "train_resnet.py", line 103, in main
    run_training()
  File "train_resnet.py", line 77, in run_training
    depth=FLAGS.num_features)
  File "train_resnet.py", line 45, in model
    model = res_3x3_pair(model, depth, "resnet_module_{}".format(idx + 1), training_switch, keep_prob_var)
  File "train_resnet.py", line 29, in res_3x3_pair
    normed = tf.nn.relu(batch_norm(data, depth, training_switch))
  File "/data/Ray/policy_network/gpu/batchnorm.py", line 22, in batch_norm
    batch_mean, batch_var = tf.nn.moments(x, [0, 1, 2], name='moments')
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/nn.py", line 712, in moments
    name=name)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/nn.py", line 649, in sufficient_statistics
    m_ss = math_ops.reduce_sum(m_ss, axes, keep_dims=keep_dims, name="mean_ss")
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 909, in reduce_sum
    keep_dims, name=name)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 2087, in _sum
    keep_dims=keep_dims, name=name)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/ops/op_def_library.py", line 704, in apply_op
    op_def=op_def)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/thouis/VENV35/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1224, in __init__
    self._traceback = _extract_stack()
</code></pre></div>
<h3 dir="auto">Environment info</h3>
<p dir="auto">Operating System: Ubuntu 16.06</p>
<p dir="auto">Installed version of CUDA and cuDNN:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="-rw-r--r-- 1 root root 189170 May 24 11:22 /usr/local/cuda/lib/libcudadevrt.a
lrwxrwxrwx 1 root root     16 May 24 11:22 /usr/local/cuda/lib/libcudart.so -&gt; libcudart.so.7.5
lrwxrwxrwx 1 root root     19 May 24 11:22 /usr/local/cuda/lib/libcudart.so.7.5 -&gt; libcudart.so.7.5.18
-rwxr-xr-x 1 root root 311596 May 24 11:22 /usr/local/cuda/lib/libcudart.so.7.5.18
-rw-r--r-- 1 root root 558020 May 24 11:22 /usr/local/cuda/lib/libcudart_static.a"><pre class="notranslate"><code class="notranslate">-rw-r--r-- 1 root root 189170 May 24 11:22 /usr/local/cuda/lib/libcudadevrt.a
lrwxrwxrwx 1 root root     16 May 24 11:22 /usr/local/cuda/lib/libcudart.so -&gt; libcudart.so.7.5
lrwxrwxrwx 1 root root     19 May 24 11:22 /usr/local/cuda/lib/libcudart.so.7.5 -&gt; libcudart.so.7.5.18
-rwxr-xr-x 1 root root 311596 May 24 11:22 /usr/local/cuda/lib/libcudart.so.7.5.18
-rw-r--r-- 1 root root 558020 May 24 11:22 /usr/local/cuda/lib/libcudart_static.a
</code></pre></div>
<p dir="auto">If installed from binary pip package, provide:</p>
<p dir="auto">pip installed from May 25 nightly build of Linux GPU version.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python -c &quot;import tensorflow; print(tensorflow.__version__)&quot;
0.8.0"><pre class="notranslate"><code class="notranslate">python -c "import tensorflow; print(tensorflow.__version__)"
0.8.0
</code></pre></div>
<h3 dir="auto">Steps to reproduce</h3>
<ul dir="auto">
<li>grab python files from <a href="https://gist.github.com/thouis/9bbd330a153b1f553fd58743a3ae4c9a">https://gist.github.com/thouis/9bbd330a153b1f553fd58743a3ae4c9a</a></li>
<li><code class="notranslate">python -u -i train_resnet.py --learning_rate 0.005  --num_modules 16 --num_features 192 --summary_dir Workspace --checkpoint_dir Workspace --batch_size 64</code></li>
</ul>
<h3 dir="auto">What have you tried?</h3>
<ol dir="auto">
<li>I tried to cast the values in _safe_shape_div() to int64 in case that was the underlying cause, but that seemed to cause different problems.</li>
</ol>
<h3 dir="auto">Logs or other output that would be helpful</h3>
<p dir="auto">All files (python, run log) at:<br>
<a href="https://gist.github.com/thouis/9bbd330a153b1f553fd58743a3ae4c9a">https://gist.github.com/thouis/9bbd330a153b1f553fd58743a3ae4c9a</a></p>
<p dir="auto">I apologize if this is not actually a issue. When trying to put tf.reduce_mean() on the gpu, I get the following error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'gradients/Mean_grad/mod': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available
         [[Node: gradients/Mean_grad/mod = Mod[T=DT_INT32, _device=&quot;/device:GPU:0&quot;](gradients/Mean_grad/add, gradients/Mean_grad/Size)]]"><pre class="notranslate"><code class="notranslate">tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'gradients/Mean_grad/mod': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available
         [[Node: gradients/Mean_grad/mod = Mod[T=DT_INT32, _device="/device:GPU:0"](gradients/Mean_grad/add, gradients/Mean_grad/Size)]]
</code></pre></div>
<p dir="auto">I am on commit <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/tensorflow/tensorflow/commit/43a8c49f04a485ffc33b510c27eda77f8f38377c/hovercard" href="https://github.com/tensorflow/tensorflow/commit/43a8c49f04a485ffc33b510c27eda77f8f38377c"><tt>43a8c49</tt></a> from Tue 5/24/2016.</p>
<p dir="auto">You can reproduce this by modifying the mnist/convolutional.py file by putting the tf.reduce_mean() and optimizer.minimize() inside "with tf.device('/gpu:0'):" statements.</p>
<p dir="auto">I am trying to do gradient calculations on their respective gpu towers, but compute_gradients() gets hung up on tf.reduce_mean().</p>
<p dir="auto">Thank you,<br>
Mark</p>