<p dir="auto">I'm running my ResNet-32 benchmark model. My code looks quite similar with the code in TensorFlow GitHub:</p>
<p dir="auto"><a href="https://github.com/tensorflow/tensorflow/tree/master/tensorflow/models/image/cifar10">https://github.com/tensorflow/tensorflow/tree/master/tensorflow/models/image/cifar10</a></p>
<p dir="auto">but the model is changed into my own ResNet-32 implementation.</p>
<p dir="auto">I'm measuring how long it takes to train a minibatch with size 128, and here are brief results. I used one GPU, 64bit Ubuntu 14.04 and Python 2.7 for all experiments.</p>
<table role="table">
<thead>
<tr>
<th align="center">TensorFlow Version</th>
<th align="center">CUDA/cuDNN Version</th>
<th align="center">GPU</th>
<th align="center">Elapsed Time (ms)</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center">0.9.0 (pip installed)</td>
<td align="center">7.5 / 4.0.7</td>
<td align="center">GTX Titan X</td>
<td align="center">75</td>
</tr>
<tr>
<td align="center">0.9.0 (<a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/tensorflow/tensorflow/commit/745f16f790d2f9abf2c6a70d6b383b152cca3cff/hovercard" href="https://github.com/tensorflow/tensorflow/commit/745f16f790d2f9abf2c6a70d6b383b152cca3cff"><tt>745f16f</tt></a>)</td>
<td align="center">8.0 RC / 5.0.5</td>
<td align="center">GTX 1080</td>
<td align="center">330</td>
</tr>
<tr>
<td align="center">0.8.0 (<a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/tensorflow/tensorflow/commit/ea9e00a630f91a459dd5858cb22e8cd1a666ba4e/hovercard" href="https://github.com/tensorflow/tensorflow/commit/ea9e00a630f91a459dd5858cb22e8cd1a666ba4e"><tt>ea9e00a</tt></a>)</td>
<td align="center">8.0 RC / 5.0.5</td>
<td align="center">GTX 1080</td>
<td align="center">60</td>
</tr>
</tbody>
</table>
<p dir="auto">The 0.8.0 version commit contains the first support for CUDA 8.0 RC as far as I know and I found this to use my GTX 1080 without performance drop. Now I can use it with reasonable performance, but I want to figure out why the 0.9.0 compiled version is 4 times slower than before.</p>
<p dir="auto">Nightly built, python2 gpu, cuda 7.5, cudnn 4.0.7, archlinux. TitanX.<br>
I run <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/batch_norm_benchmark.py">batch_norm_benchmark.py</a> and got the following output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Forward convolution (lower layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.043865 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.053864 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.088060 secs
=== op vs py: 22.8% ===
=== py vs slow: 63.5% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.287913 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.284179 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.287409 secs
=== op vs py: -1.3% ===
=== py vs slow: 1.1% ===
Forward/backward convolution (lower layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:True - 0.220112 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.201284 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.295103 secs
=== op vs py: -8.6% ===
=== py vs slow: 46.6% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:True - 2.108785 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.578407 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.863753 secs
=== op vs py: -59.0% ===
=== py vs slow: -96.6% ===
Forward convolution (higher layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.025443 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.033344 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.048162 secs
=== op vs py: 31.1% ===
=== py vs slow: 44.4% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.164241 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.161473 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.163212 secs
=== op vs py: -1.7% ===
=== py vs slow: 1.1% ===
Forward/backward convolution (higher layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:True - 0.111931 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.118556 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.163289 secs
=== op vs py: 5.9% ===
=== py vs slow: 37.7% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:True - 1.194288 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.329403 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.491005 secs
=== op vs py: -72.4% ===
=== py vs slow: 49.1% ===
Forward fully-connected.
cpu shape:2/1 #layers:10 mode:py scale:True train:False - 0.001866 secs
cpu shape:2/1 #layers:10 mode:slow scale:True train:False - 0.002859 secs
=== py vs slow: 53.2% ===
gpu shape:2/1 #layers:10 mode:py scale:True train:False - 0.002420 secs
gpu shape:2/1 #layers:10 mode:slow scale:True train:False - 0.002523 secs
=== py vs slow: 4.3% ===
Forward/backward fully-connected.
cpu shape:2/1 #layers:10 mode:py scale:True train:True - 0.004973 secs
cpu shape:2/1 #layers:10 mode:slow scale:True train:True - 0.006781 secs
=== py vs slow: 36.4% ===
gpu shape:2/1 #layers:10 mode:py scale:True train:True - 0.006370 secs
gpu shape:2/1 #layers:10 mode:slow scale:True train:True - 0.008182 secs
=== py vs slow: 28.4% ==="><pre class="notranslate"><code class="notranslate">Forward convolution (lower layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.043865 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.053864 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.088060 secs
=== op vs py: 22.8% ===
=== py vs slow: 63.5% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.287913 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.284179 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.287409 secs
=== op vs py: -1.3% ===
=== py vs slow: 1.1% ===
Forward/backward convolution (lower layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:True - 0.220112 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.201284 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.295103 secs
=== op vs py: -8.6% ===
=== py vs slow: 46.6% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:True - 2.108785 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.578407 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.863753 secs
=== op vs py: -59.0% ===
=== py vs slow: -96.6% ===
Forward convolution (higher layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.025443 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.033344 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.048162 secs
=== op vs py: 31.1% ===
=== py vs slow: 44.4% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.164241 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.161473 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.163212 secs
=== op vs py: -1.7% ===
=== py vs slow: 1.1% ===
Forward/backward convolution (higher layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:True - 0.111931 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.118556 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.163289 secs
=== op vs py: 5.9% ===
=== py vs slow: 37.7% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:True - 1.194288 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.329403 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.491005 secs
=== op vs py: -72.4% ===
=== py vs slow: 49.1% ===
Forward fully-connected.
cpu shape:2/1 #layers:10 mode:py scale:True train:False - 0.001866 secs
cpu shape:2/1 #layers:10 mode:slow scale:True train:False - 0.002859 secs
=== py vs slow: 53.2% ===
gpu shape:2/1 #layers:10 mode:py scale:True train:False - 0.002420 secs
gpu shape:2/1 #layers:10 mode:slow scale:True train:False - 0.002523 secs
=== py vs slow: 4.3% ===
Forward/backward fully-connected.
cpu shape:2/1 #layers:10 mode:py scale:True train:True - 0.004973 secs
cpu shape:2/1 #layers:10 mode:slow scale:True train:True - 0.006781 secs
=== py vs slow: 36.4% ===
gpu shape:2/1 #layers:10 mode:py scale:True train:True - 0.006370 secs
gpu shape:2/1 #layers:10 mode:slow scale:True train:True - 0.008182 secs
=== py vs slow: 28.4% ===
</code></pre></div>
<p dir="auto">Although <code class="notranslate">sess.run</code> in a loop might not be an accurate way to benchmark, I think the performance regression exists because running the same script with an earlier binary built (not sure which commit it is) is 10x-20x faster.</p>