<p dir="auto">Use tf.one_hot() on Windows 10, GPU, Nvidia 970.  Error CUDA_ERROR_LAUNCH_FAILED happens.</p>
<h3 dir="auto">What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?</h3>
<p dir="auto">I have post an issue in <a href="http://stackoverflow.com/questions/41115476/tensorflow-gpu-cuda-error-launch-failed-on-tf-one-hot" rel="nofollow">http://stackoverflow.com/questions/41115476/tensorflow-gpu-cuda-error-launch-failed-on-tf-one-hot</a> .  No solution found.  More people report the same issue.</p>
<h3 dir="auto">Environment info</h3>
<ul dir="auto">
<li>TensorFlow 0.12.0-rc1</li>
<li>Python 3.5</li>
<li>CUDA 8.0</li>
<li>cuDNN 5.1</li>
<li>OS: Windows 10</li>
<li>GPU: GeForce GTX 970</li>
</ul>
<h3 dir="auto">If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf
idx_0 = tf.placeholder(tf.int64, [None])
mask = tf.one_hot(idx_0, 3, axis=-1)
sess = tf.Session()
sess.run(tf.global_variables_initializer())
a = sess.run([mask],feed_dict={idx_0:[0,1,2]})
print(a)"><pre class="notranslate"><code class="notranslate">import tensorflow as tf
idx_0 = tf.placeholder(tf.int64, [None])
mask = tf.one_hot(idx_0, 3, axis=-1)
sess = tf.Session()
sess.run(tf.global_variables_initializer())
a = sess.run([mask],feed_dict={idx_0:[0,1,2]})
print(a)
</code></pre></div>
<h3 dir="auto">What other attempted solutions have you tried?</h3>
<ol dir="auto">
<li>Run the code in same machine, Linux, GPU.  No error.</li>
<li>Run the code in same machine, Windows, CPU.  No error.</li>
</ol>
<h3 dir="auto">Logs or other output that would be helpful</h3>
<p dir="auto">(If logs are large, please upload as attachment or provide link).</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_driver.cc:1177] could not synchronize on CUDA context: CUDA_ERROR_LAUNCH_FAILED :: No stack trace available
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_LAUNCH_FAILED
F c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_util.cc:370] GPU sync failed"><pre class="notranslate"><code class="notranslate">E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_driver.cc:1177] could not synchronize on CUDA context: CUDA_ERROR_LAUNCH_FAILED :: No stack trace available
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_LAUNCH_FAILED
F c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_util.cc:370] GPU sync failed
</code></pre></div>
<p dir="auto">I'm using a LeNet-5 mnist example from Udacity's course. Link to the source code is below.<br>
Training works ok on a CPU (config = tf.ConfigProto(device_count = {'GPU': 0})),<br>
but fails in a GPU mode with the following 'CUDA_ERROR_ILLEGAL_ADDRESS' error:</p>
<blockquote>
<p dir="auto">I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:885] Found device 0 with properties:<br>
name: GeForce GTX 1060 6GB<br>
major: 6 minor: 1 memoryClockRate (GHz) 1.7845<br>
pciBusID 0000:01:00.0<br>
Total memory: 6.00GiB<br>
Free memory: 5.01GiB<br>
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:906] DMA: 0<br>
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:916] 0:   Y<br>
I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -&gt; (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0)<br>
Training...<br>
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_event.cc:49] Error polling for event status: failed to query event: <strong>CUDA_ERROR_ILLEGAL_ADDRESS</strong><br>
F c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_event_mgr.cc:198] Unexpected Event status: 1</p>
</blockquote>
<p dir="auto">I have reproduced same error on two setups.<br>
Environment 1 (Home PC):</p>
<ul dir="auto">
<li>windows 10;</li>
<li>latest anaconda 4.2.0, python 3.5;</li>
<li>cuda 8</li>
<li>cudnn 5.1 (for win10)</li>
<li>tensorflow 0.12.0 gpu (<a href="https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-0.12.0-cp35-cp35m-win_amd64.whl" rel="nofollow">https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-0.12.0-cp35-cp35m-win_amd64.whl</a>)</li>
<li>GeForce GTX 1060 6Gb</li>
</ul>
<p dir="auto">Environment 2 (Work PC):</p>
<ul dir="auto">
<li>windows 7;</li>
<li>latest anaconda 4.2.0, python 3.5;</li>
<li>cuda 8</li>
<li>cudnn 5.1 (for win7)</li>
<li>tensorflow 0.12.0 gpu (<a href="https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-0.12.0-cp35-cp35m-win_amd64.whl" rel="nofollow">https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-0.12.0-cp35-cp35m-win_amd64.whl</a>)</li>
<li>GeForce GTX 660 3Gb</li>
</ul>
<p dir="auto">I'm sharing two scripts with minor changes that allow a workaround:<br>
<a href="https://drive.google.com/open?id=0B6jkkqMOGy5cNHh3TVpxU283Ykk" rel="nofollow">https://drive.google.com/open?id=0B6jkkqMOGy5cNHh3TVpxU283Ykk</a></p>
<p dir="auto">Main difference:<br>
Script from example that crashes (LabLenetBad.py) uses raw mnist label data with the tf.one_hot() call.<br>
The workaround (LabLenetGood.py) reads mnist data with (one_hot=True) flag and does not use tf.one_hot() call on the Y placeholder.</p>
<p dir="auto">I think that tf.one_hot does not work properly on the gpu.</p>