<h3 dir="auto">Environment info</h3>
<p dir="auto">Operating System: Ubuntu 14.04, AWS g2.2xlarge</p>
<p dir="auto">Installed version of CUDA and cuDNN:  CUDA 7.5, cuDNN 4</p>
<p dir="auto">Tried 3 different version using pip package:<br>
0.8.0:<br>
sudo pip install --upgrade <a href="https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.8.0-cp27-none-linux_x86_64.whl" rel="nofollow">https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.8.0-cp27-none-linux_x86_64.whl</a></p>
<p dir="auto">0.9.0:<br>
sudo pip install --upgrade <a href="https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.9.0-cp27-none-linux_x86_64.whl" rel="nofollow">https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.9.0-cp27-none-linux_x86_64.whl</a></p>
<p dir="auto">0.10.0 rc<br>
sudo pip install --upgrade <a href="https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.10.0rc0-cp27-none-linux_x86_64.whl" rel="nofollow">https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.10.0rc0-cp27-none-linux_x86_64.whl</a></p>
<h3 dir="auto">If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)</h3>
<p dir="auto">On the same machine, install different version of tf and run the <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/models/image/cifar10/cifar10_train.py">cifar10_train.py</a></p>
<h3 dir="auto">Logs or other output that would be helpful</h3>
<p dir="auto">For version 0.8.0 and 0.9.0, the GPU usage is about 30%</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="+------------------------------------------------------+                       
| NVIDIA-SMI 352.99     Driver Version: 352.99         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID K520           Off  | 0000:00:03.0     Off |                  N/A |
| N/A   37C    P0    51W / 125W |   3854MiB /  4095MiB |     35%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0     16558    C   python                                        3841MiB |
+-----------------------------------------------------------------------------+"><pre class="notranslate"><code class="notranslate">+------------------------------------------------------+                       
| NVIDIA-SMI 352.99     Driver Version: 352.99         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID K520           Off  | 0000:00:03.0     Off |                  N/A |
| N/A   37C    P0    51W / 125W |   3854MiB /  4095MiB |     35%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0     16558    C   python                                        3841MiB |
+-----------------------------------------------------------------------------+
</code></pre></div>
<p dir="auto">But the speed is about ~0.24 sec/batch for version 0.8.0 and ~0.22 sec/batch for version 0.9.0</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2016-08-26 21:20:41.309814: step 230, loss = 4.28 (522.0 examples/sec; 0.245 sec/batch)
2016-08-26 21:20:43.625017: step 240, loss = 4.26 (582.3 examples/sec; 0.220 sec/batch)
2016-08-26 21:20:45.953772: step 250, loss = 4.24 (544.4 examples/sec; 0.235 sec/batch)
2016-08-26 21:20:48.302202: step 260, loss = 4.23 (540.1 examples/sec; 0.237 sec/batch)
2016-08-26 21:20:50.643760: step 270, loss = 4.21 (554.6 examples/sec; 0.231 sec/batch)
2016-08-26 21:20:52.955326: step 280, loss = 4.20 (545.6 examples/sec; 0.235 sec/batch)
2016-08-26 21:20:55.399758: step 290, loss = 4.18 (476.4 examples/sec; 0.269 sec/batch)
2016-08-26 21:20:57.825254: step 300, loss = 4.17 (548.0 examples/sec; 0.234 sec/batch)
2016-08-26 21:21:00.453533: step 310, loss = 4.15 (543.6 examples/sec; 0.235 sec/batch)
2016-08-26 21:21:02.876055: step 320, loss = 4.14 (513.9 examples/sec; 0.249 sec/batch)
2016-08-26 21:21:05.229421: step 330, loss = 4.13 (580.3 examples/sec; 0.221 sec/batch)
2016-08-26 21:21:07.614095: step 340, loss = 4.11 (528.6 examples/sec; 0.242 sec/batch)"><pre class="notranslate"><code class="notranslate">2016-08-26 21:20:41.309814: step 230, loss = 4.28 (522.0 examples/sec; 0.245 sec/batch)
2016-08-26 21:20:43.625017: step 240, loss = 4.26 (582.3 examples/sec; 0.220 sec/batch)
2016-08-26 21:20:45.953772: step 250, loss = 4.24 (544.4 examples/sec; 0.235 sec/batch)
2016-08-26 21:20:48.302202: step 260, loss = 4.23 (540.1 examples/sec; 0.237 sec/batch)
2016-08-26 21:20:50.643760: step 270, loss = 4.21 (554.6 examples/sec; 0.231 sec/batch)
2016-08-26 21:20:52.955326: step 280, loss = 4.20 (545.6 examples/sec; 0.235 sec/batch)
2016-08-26 21:20:55.399758: step 290, loss = 4.18 (476.4 examples/sec; 0.269 sec/batch)
2016-08-26 21:20:57.825254: step 300, loss = 4.17 (548.0 examples/sec; 0.234 sec/batch)
2016-08-26 21:21:00.453533: step 310, loss = 4.15 (543.6 examples/sec; 0.235 sec/batch)
2016-08-26 21:21:02.876055: step 320, loss = 4.14 (513.9 examples/sec; 0.249 sec/batch)
2016-08-26 21:21:05.229421: step 330, loss = 4.13 (580.3 examples/sec; 0.221 sec/batch)
2016-08-26 21:21:07.614095: step 340, loss = 4.11 (528.6 examples/sec; 0.242 sec/batch)
</code></pre></div>
<hr>
<p dir="auto">For 0.10.0.rc0, the GPU usage is ~90％：</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="+------------------------------------------------------+                       
| NVIDIA-SMI 352.99     Driver Version: 352.99         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID K520           Off  | 0000:00:03.0     Off |                  N/A |
| N/A   35C    P0    46W / 125W |   3818MiB /  4095MiB |     90%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0     16702    C   python                                        3805MiB |
+-----------------------------------------------------------------------------+"><pre class="notranslate"><code class="notranslate">+------------------------------------------------------+                       
| NVIDIA-SMI 352.99     Driver Version: 352.99         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID K520           Off  | 0000:00:03.0     Off |                  N/A |
| N/A   35C    P0    46W / 125W |   3818MiB /  4095MiB |     90%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0     16702    C   python                                        3805MiB |
+-----------------------------------------------------------------------------+
</code></pre></div>
<p dir="auto">But the speed is only ~0.34 sec/batch:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="016-08-26 21:24:11.512601: step 30, loss = 4.38 (371.3 examples/sec; 0.345 sec/batch)
2016-08-26 21:24:14.875387: step 40, loss = 4.42 (379.0 examples/sec; 0.338 sec/batch)
2016-08-26 21:24:18.248093: step 50, loss = 4.27 (368.0 examples/sec; 0.348 sec/batch)
2016-08-26 21:24:21.609797: step 60, loss = 4.24 (379.8 examples/sec; 0.337 sec/batch)
2016-08-26 21:24:24.987058: step 70, loss = 4.25 (376.4 examples/sec; 0.340 sec/batch)
2016-08-26 21:24:28.387080: step 80, loss = 4.38 (381.0 examples/sec; 0.336 sec/batch)
2016-08-26 21:24:31.775519: step 90, loss = 4.18 (377.8 examples/sec; 0.339 sec/batch)"><pre class="notranslate"><code class="notranslate">016-08-26 21:24:11.512601: step 30, loss = 4.38 (371.3 examples/sec; 0.345 sec/batch)
2016-08-26 21:24:14.875387: step 40, loss = 4.42 (379.0 examples/sec; 0.338 sec/batch)
2016-08-26 21:24:18.248093: step 50, loss = 4.27 (368.0 examples/sec; 0.348 sec/batch)
2016-08-26 21:24:21.609797: step 60, loss = 4.24 (379.8 examples/sec; 0.337 sec/batch)
2016-08-26 21:24:24.987058: step 70, loss = 4.25 (376.4 examples/sec; 0.340 sec/batch)
2016-08-26 21:24:28.387080: step 80, loss = 4.38 (381.0 examples/sec; 0.336 sec/batch)
2016-08-26 21:24:31.775519: step 90, loss = 4.18 (377.8 examples/sec; 0.339 sec/batch)
</code></pre></div>
<h3 dir="auto">Environment info</h3>
<p dir="auto">Operating System:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Linux 4.4.11-23.53"><pre class="notranslate"><code class="notranslate">Linux 4.4.11-23.53
</code></pre></div>
<h3 dir="auto">Observations</h3>
<p dir="auto">I am testing on a Tesla K80 (details below) using the following lines:</p>
<ul dir="auto">
<li><code class="notranslate">nvidia-docker run --rm -it -v /tmp/cifar10_data:/tmp/cifar10_data tensorflow/tensorflow:0.9.0-devel-gpu  bash -c 'ln -s /usr/local/nvidia/lib64/libcuda.so.1 /usr/lib/x86_64-linux-gnu/libcuda.so &amp;&amp; python /usr/local/lib/python2.7/dist-packages/tensorflow/models/image/cifar10/cifar10_multi_gpu_train.py'</code></li>
<li><code class="notranslate">nvidia-docker run --rm -it -v /tmp/cifar10_data:/tmp/cifar10_data tensorflow/tensorflow:0.10.0rc0-devel-gpu python /usr/local/lib/python2.7/dist-packages/tensorflow/models/image/cifar10/cifar10_multi_gpu_train.py</code></li>
</ul>
<p dir="auto">On r0.0.9, I get: ~<code class="notranslate">575 examples/sec</code>.<br>
On r0.10.0rc0 I get ~<code class="notranslate">425 examples/sec</code>.</p>
<p dir="auto">Here are the device stats:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:04.0
Total memory: 11.17GiB
Free memory: 11.11GiB"><pre class="notranslate"><code class="notranslate">major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:04.0
Total memory: 11.17GiB
Free memory: 11.11GiB
</code></pre></div>
<p dir="auto">I have the: Nvidia Driver Version: 367.35</p>