<p dir="auto">I am running Tensor-flow on Google compute engine instances and I met with this problem when the training period is finished (after step 9990).</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2017-03-17 00:24:23.021298: step 9980, loss = 0.72 (242.4 examples/sec; 0.264 sec/batch)
2017-03-17 00:24:34.693612: step 9990, loss = 0.79 (208.4 examples/sec; 0.307 sec/batch)
E tensorflow/stream_executor/cuda/cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_DEINITIALIZED
F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:198] Unexpected Event status: 1
Aborted (core dumped)"><pre class="notranslate"><code class="notranslate">2017-03-17 00:24:23.021298: step 9980, loss = 0.72 (242.4 examples/sec; 0.264 sec/batch)
2017-03-17 00:24:34.693612: step 9990, loss = 0.79 (208.4 examples/sec; 0.307 sec/batch)
E tensorflow/stream_executor/cuda/cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_DEINITIALIZED
F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:198] Unexpected Event status: 1
Aborted (core dumped)
</code></pre></div>
<p dir="auto">Can anyone identify this problem and let me know how to fix it?</p>
<h2 dir="auto">Environment info</h2>
<h3 dir="auto">Hardware Platform:</h3>
<p dir="auto">Google Cloud Compute Engine<br>
4 vCPUs, 15 GB memory<br>
4 * NVIDIA Tesla K80</p>
<h3 dir="auto">Software Platform:</h3>
<p dir="auto">Operating System: Ubuntu 16.04<br>
cuDNN v5<br>
CUDA toolkit 8.0<br>
CUDA Compute Capability 3.7</p>
<h3 dir="auto">Tensor-flow:</h3>
<p dir="auto">Installed from pip<br>
<a href="https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.0.1-cp34-cp34m-linux_x86_64.whl" rel="nofollow">https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.0.1-cp34-cp34m-linux_x86_64.whl</a><br>
(Installing from source also doesn't work)</p>
<h2 dir="auto">Steps to reproduce</h2>
<p dir="auto">python3 cifar10_multi_gpu_train.py --num_gpus=4</p>
<p dir="auto">Error occurs after training.</p>
<h2 dir="auto">What have you tried?</h2>
<ol dir="auto">
<li>restart machine and rerun several times;</li>
<li>--num_gpus=2 --&gt; Segmentation fault</li>
<li>Reinstall Tensor-flow from source</li>
</ol>
<h2 dir="auto">Logs or other output that would be helpful</h2>
<p dir="auto">Log file is attached.<br>
<a href="https://github.com/tensorflow/tensorflow/files/852192/Terminal.Saved.Output.txt">Terminal Saved Output.txt</a></p>