<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: Yes</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Linux Ubuntu 16.04.5</li>
<li><strong>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device</strong>: N/A</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: Issue occurs on both pip binary and compiled from source</li>
<li><strong>TensorFlow version (use command below)</strong>: 'v1.8.0-8-g23c2187' 1.8.0</li>
<li><strong>Python version</strong>: 3.5.2</li>
<li><strong>Bazel version (if compiling from source)</strong>: 0.16.1</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>: gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609</li>
<li><strong>CUDA/cuDNN version</strong>: CUDA 9.0, cuDNN 7.2.1.38</li>
<li><strong>GPU model and memory</strong>: GeForce GTX 1080, 11178MiB</li>
<li><strong>Exact command to reproduce</strong>: Cannot share the exact codebase</li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">Tensorflow attempts to swap-out a node to the CPU to save memory, even when the node is already on CPU.</p>
<p dir="auto">This happens in <code class="notranslate">core/grappler/optimizers/memory_optimizers.cc</code> with the op <code class="notranslate">_CopyFromGpuToHost</code>.<br>
But since this swapping is intended to save GPU memory by swapping temporarily nodes from GPU to the the CPU, the kernel for that op in <code class="notranslate">core/grappler/optimizers/gpu_swapping_kernels.cc</code> is only for GPU devices.</p>
<p dir="auto">A suggested fix will be to simply check if the node to be swapped out is already in CPU before making it eligible for swap_out/swap_in. Note that on reducing the model size this error disappears as more memory becomes free.</p>
<h4 dir="auto">Some more information confirming the issue -</h4>
<p dir="auto">The op is correctly registered in OpRegistry as shown by <code class="notranslate">tensorflow::OpRegistry::Global()-&gt;DebugString(false)</code>.<br>
The OpKernel <code class="notranslate">CopyFromGpuToHostKernel</code> is built according to bazel logs.<br>
The OpKernel is loaded on importing tensorflow in python, verified by adding a print in that file using <code class="notranslate">__attribute__((constructor)) )</code></p>
<h3 dir="auto">Source code / logs</h3>
<p dir="auto">The relevant error is -</p>
<p dir="auto">tensorflow.python.framework.errors_impl.NotFoundError: No registered '_CopyFromGpuToHost' OpKernel for CPU devices compatible with node swap_out_context_zoom/GatherNd_2_0 = _CopyFromGpuToHost<a href="_arg_context_sents_len_0_10">T=DT_INT32, _class=["loc@context_zoom/GatherNd_2_0"], _device="/job:localhost/replica:0/task:0/device:CPU:0"</a><br>
.  Registered:  device='GPU'</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" [[Node: swap_out_context_zoom/GatherNd_2_0 = _CopyFromGpuToHost[T=DT_INT32, _class=[&quot;loc@context_zoom/GatherNd_2_0&quot;], _device=&quot;/job:localhost/replica:0/task:0/device:CPU:0&quot;](_arg_context_sents_len_0_10)"><pre class="notranslate"><code class="notranslate"> [[Node: swap_out_context_zoom/GatherNd_2_0 = _CopyFromGpuToHost[T=DT_INT32, _class=["loc@context_zoom/GatherNd_2_0"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_context_sents_len_0_10)
</code></pre></div>
<p dir="auto">I have attached the full log here -<br>
<a href="https://gist.github.com/akhilkedia/c13ff191e08b4462a3870a0b3a091e93">https://gist.github.com/akhilkedia/c13ff191e08b4462a3870a0b3a091e93</a></p>
<p dir="auto">Please go to Stack Overflow for help and support:</p>
<p dir="auto"><a href="https://stackoverflow.com/questions/tagged/tensorflow" rel="nofollow">https://stackoverflow.com/questions/tagged/tensorflow</a></p>
<p dir="auto">If you open a GitHub issue, here is our policy:</p>
<ol dir="auto">
<li>It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).</li>
<li>The form below must be filled out.</li>
<li>It shouldn't be a TensorBoard issue. Those go <a href="https://github.com/tensorflow/tensorboard/issues">here</a>.</li>
</ol>
<p dir="auto"><strong>Here's why we have that policy</strong>: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.</p>
<hr>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: Yes. Added some custom TF operators.</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Ubuntu 18.04</li>
<li><strong>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device</strong>: N/A</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: Binary</li>
<li><strong>TensorFlow version (use command below)</strong>:1.10.1</li>
<li><strong>Python version</strong>:3</li>
<li><strong>Bazel version (if compiling from source)</strong>:</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>:</li>
<li><strong>CUDA/cuDNN version</strong>:9.2</li>
<li><strong>GPU model and memory</strong>: GeForce GTX 970</li>
<li><strong>Exact command to reproduce</strong>:python3 train/train.py --gpu 0 --model frustum_pointnets_v2 --log_dir train/log_v2 --num_point 1024 --max_epoch 201 --batch_size 24 --decay_step 800000 --decay_rate 0.5</li>
</ul>
<p dir="auto">You can collect some of this information using our environment capture script:</p>
<p dir="auto"><a href="https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh">https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh</a></p>
<p dir="auto">You can obtain the TensorFlow version with</p>
<p dir="auto">python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"</p>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">Getting an error that "No registered '_CopyFromGpuToHost' OpKernel". Not able to find the refernce to this OpKernel anywhere.</p>
<h3 dir="auto">Source code / logs</h3>
<p dir="auto">pid: 16638<br>
WARNING:tensorflow:From /home/rishi/sw/frustum-pointnets/models/pointnet_util.py:126: calling reduce_max (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.<br>
Instructions for updating:<br>
keep_dims is deprecated, use keepdims instead<br>
WARNING:tensorflow:From /home/rishi/sw/frustum-pointnets/models/pointnet_util.py:212: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.<br>
Instructions for updating:<br>
keep_dims is deprecated, use keepdims instead<br>
WARNING:tensorflow:From /home/rishi/.local/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py:519: calling conv1d (from tensorflow.python.ops.nn_ops) with data_format=NHWC is deprecated and will be removed in a future version.<br>
Instructions for updating:<br>
<code class="notranslate">NHWC</code> for data_format is deprecated, use <code class="notranslate">NWC</code> instead<br>
2018-09-06 19:00:38.848283: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA<br>
2018-09-06 19:00:38.920714: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero<br>
2018-09-06 19:00:38.921052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:<br>
name: GeForce GTX 970 major: 5 minor: 2 memoryClockRate(GHz): 1.1775<br>
pciBusID: 0000:01:00.0<br>
totalMemory: 3.94GiB freeMemory: 3.56GiB<br>
2018-09-06 19:00:38.921067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0<br>
2018-09-06 19:00:39.115930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:<br>
2018-09-06 19:00:39.115958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0<br>
2018-09-06 19:00:39.115967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N<br>
2018-09-06 19:00:39.116154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3282 MB memory) -&gt; physical GPU (device: 0, name: GeForce GTX 970, pci bus id: 0000:01:00.0, compute capability: 5.2)<br>
**** EPOCH 000 ****<br>
2018-09-06 19:00:42.022472<br>
2018-09-06 19:00:42.305938: W tensorflow/core/framework/allocator.cc:108] Allocation of 50331648 exceeds 10% of system memory.<br>
2018-09-06 19:00:42.322253: W tensorflow/core/framework/allocator.cc:108] Allocation of 50331648 exceeds 10% of system memory.<br>
2018-09-06 19:00:42.337831: W tensorflow/core/framework/allocator.cc:108] Allocation of 100663296 exceeds 10% of system memory.<br>
2018-09-06 19:00:42.368540: W tensorflow/core/framework/allocator.cc:108] Allocation of 100663296 exceeds 10% of system memory.<br>
2018-09-06 19:00:42.401274: W tensorflow/core/framework/allocator.cc:108] Allocation of 50331648 exceeds 10% of system memory.<br>
2018-09-06 19:00:52.106763: E tensorflow/core/common_runtime/executor.cc:697] Executor failed to create kernel. Not found: No registered '_CopyFromGpuToHost' OpKernel for CPU devices compatible with node swap_out_gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2 = _CopyFromGpuToHost<a href="fa_layer3/truediv_2/_91">T=DT_FLOAT, _class=["loc@gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2"], _device="/job:localhost/replica:0/task:0/device:CPU:0"</a><br>
.  Registered:  device='GPU'</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" [[Node: swap_out_gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2 = _CopyFromGpuToHost[T=DT_FLOAT, _class=[&quot;loc@gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2&quot;], _device=&quot;/job:localhost/replica:0/task:0/device:CPU:0&quot;](fa_layer3/truediv_2/_91)]]"><pre class="notranslate"><code class="notranslate"> [[Node: swap_out_gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2 = _CopyFromGpuToHost[T=DT_FLOAT, _class=["loc@gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](fa_layer3/truediv_2/_91)]]
</code></pre></div>
<p dir="auto">Traceback (most recent call last):<br>
File "/home/rishi/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1278, in _do_call<br>
return fn(*args)<br>
File "/home/rishi/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1263, in _run_fn<br>
options, feed_dict, fetch_list, target_list, run_metadata)<br>
File "/home/rishi/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun<br>
run_metadata)<br>
tensorflow.python.framework.errors_impl.NotFoundError: No registered '_CopyFromGpuToHost' OpKernel for CPU devices compatible with node swap_out_gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2 = _CopyFromGpuToHost<a href="fa_layer3/truediv_2/_91">T=DT_FLOAT, _class=["loc@gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2"], _device="/job:localhost/replica:0/task:0/device:CPU:0"</a><br>
.  Registered:  device='GPU'</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" [[Node: swap_out_gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2 = _CopyFromGpuToHost[T=DT_FLOAT, _class=[&quot;loc@gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2&quot;], _device=&quot;/job:localhost/replica:0/task:0/device:CPU:0&quot;](fa_layer3/truediv_2/_91)]]"><pre class="notranslate"><code class="notranslate"> [[Node: swap_out_gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2 = _CopyFromGpuToHost[T=DT_FLOAT, _class=["loc@gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](fa_layer3/truediv_2/_91)]]
</code></pre></div>
<p dir="auto">During handling of the above exception, another exception occurred:</p>
<p dir="auto">Traceback (most recent call last):<br>
File "train/train.py", line 376, in <br>
train()<br>
File "train/train.py", line 201, in train<br>
train_one_epoch(sess, ops, train_writer)<br>
File "train/train.py", line 256, in train_one_epoch<br>
feed_dict=feed_dict)<br>
File "/home/rishi/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 877, in run<br>
run_metadata_ptr)<br>
File "/home/rishi/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1100, in _run<br>
feed_dict_tensor, options, run_metadata)<br>
File "/home/rishi/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run<br>
run_metadata)<br>
File "/home/rishi/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call<br>
raise type(e)(node_def, op, message)<br>
tensorflow.python.framework.errors_impl.NotFoundError: No registered '_CopyFromGpuToHost' OpKernel for CPU devices compatible with node swap_out_gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2 = _CopyFromGpuToHost<a href="fa_layer3/truediv_2/_91">T=DT_FLOAT, _class=["loc@gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2"], _device="/job:localhost/replica:0/task:0/device:CPU:0"</a><br>
.  Registered:  device='GPU'</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" [[Node: swap_out_gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2 = _CopyFromGpuToHost[T=DT_FLOAT, _class=[&quot;loc@gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2&quot;], _device=&quot;/job:localhost/replica:0/task:0/device:CPU:0&quot;](fa_layer3/truediv_2/_91)]]"><pre class="notranslate"><code class="notranslate"> [[Node: swap_out_gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2 = _CopyFromGpuToHost[T=DT_FLOAT, _class=["loc@gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](fa_layer3/truediv_2/_91)]]
</code></pre></div>