<p dir="auto">Nightly built, python2 gpu, cuda 7.5, cudnn 4.0.7, archlinux. TitanX.<br>
I run <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/batch_norm_benchmark.py">batch_norm_benchmark.py</a> and got the following output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Forward convolution (lower layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.043865 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.053864 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.088060 secs
=== op vs py: 22.8% ===
=== py vs slow: 63.5% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.287913 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.284179 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.287409 secs
=== op vs py: -1.3% ===
=== py vs slow: 1.1% ===
Forward/backward convolution (lower layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:True - 0.220112 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.201284 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.295103 secs
=== op vs py: -8.6% ===
=== py vs slow: 46.6% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:True - 2.108785 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.578407 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.863753 secs
=== op vs py: -59.0% ===
=== py vs slow: -96.6% ===
Forward convolution (higher layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.025443 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.033344 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.048162 secs
=== op vs py: 31.1% ===
=== py vs slow: 44.4% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.164241 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.161473 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.163212 secs
=== op vs py: -1.7% ===
=== py vs slow: 1.1% ===
Forward/backward convolution (higher layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:True - 0.111931 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.118556 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.163289 secs
=== op vs py: 5.9% ===
=== py vs slow: 37.7% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:True - 1.194288 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.329403 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.491005 secs
=== op vs py: -72.4% ===
=== py vs slow: 49.1% ===
Forward fully-connected.
cpu shape:2/1 #layers:10 mode:py scale:True train:False - 0.001866 secs
cpu shape:2/1 #layers:10 mode:slow scale:True train:False - 0.002859 secs
=== py vs slow: 53.2% ===
gpu shape:2/1 #layers:10 mode:py scale:True train:False - 0.002420 secs
gpu shape:2/1 #layers:10 mode:slow scale:True train:False - 0.002523 secs
=== py vs slow: 4.3% ===
Forward/backward fully-connected.
cpu shape:2/1 #layers:10 mode:py scale:True train:True - 0.004973 secs
cpu shape:2/1 #layers:10 mode:slow scale:True train:True - 0.006781 secs
=== py vs slow: 36.4% ===
gpu shape:2/1 #layers:10 mode:py scale:True train:True - 0.006370 secs
gpu shape:2/1 #layers:10 mode:slow scale:True train:True - 0.008182 secs
=== py vs slow: 28.4% ==="><pre class="notranslate"><code class="notranslate">Forward convolution (lower layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.043865 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.053864 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.088060 secs
=== op vs py: 22.8% ===
=== py vs slow: 63.5% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.287913 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.284179 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.287409 secs
=== op vs py: -1.3% ===
=== py vs slow: 1.1% ===
Forward/backward convolution (lower layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:True - 0.220112 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.201284 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.295103 secs
=== op vs py: -8.6% ===
=== py vs slow: 46.6% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:True - 2.108785 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.578407 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.863753 secs
=== op vs py: -59.0% ===
=== py vs slow: -96.6% ===
Forward convolution (higher layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.025443 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.033344 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.048162 secs
=== op vs py: 31.1% ===
=== py vs slow: 44.4% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:False - 0.164241 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:False - 0.161473 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:False - 0.163212 secs
=== op vs py: -1.7% ===
=== py vs slow: 1.1% ===
Forward/backward convolution (higher layers).
cpu shape:4/3 #layers:10 mode:op scale:True train:True - 0.111931 secs
cpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.118556 secs
cpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.163289 secs
=== op vs py: 5.9% ===
=== py vs slow: 37.7% ===
gpu shape:4/3 #layers:10 mode:op scale:True train:True - 1.194288 secs
gpu shape:4/3 #layers:10 mode:py scale:True train:True - 0.329403 secs
gpu shape:4/3 #layers:10 mode:slow scale:True train:True - 0.491005 secs
=== op vs py: -72.4% ===
=== py vs slow: 49.1% ===
Forward fully-connected.
cpu shape:2/1 #layers:10 mode:py scale:True train:False - 0.001866 secs
cpu shape:2/1 #layers:10 mode:slow scale:True train:False - 0.002859 secs
=== py vs slow: 53.2% ===
gpu shape:2/1 #layers:10 mode:py scale:True train:False - 0.002420 secs
gpu shape:2/1 #layers:10 mode:slow scale:True train:False - 0.002523 secs
=== py vs slow: 4.3% ===
Forward/backward fully-connected.
cpu shape:2/1 #layers:10 mode:py scale:True train:True - 0.004973 secs
cpu shape:2/1 #layers:10 mode:slow scale:True train:True - 0.006781 secs
=== py vs slow: 36.4% ===
gpu shape:2/1 #layers:10 mode:py scale:True train:True - 0.006370 secs
gpu shape:2/1 #layers:10 mode:slow scale:True train:True - 0.008182 secs
=== py vs slow: 28.4% ===
</code></pre></div>
<p dir="auto">Although <code class="notranslate">sess.run</code> in a loop might not be an accurate way to benchmark, I think the performance regression exists because running the same script with an earlier binary built (not sure which commit it is) is 10x-20x faster.</p>
<p dir="auto">This issue is a follow-up to <a href="http://stackoverflow.com/questions/38704095/tensorflow-rc-0-10-3x-slower-than-0-9" rel="nofollow">this posting</a> on stack overflow. As noted, the 0.10 version is ~3x slower than version 0.9 for a particular training script I am using. The same slow-down is observed when using the pip version with CuDNN4 or with a version compiled from source and CuDNN 5.1.</p>
<p dir="auto">I'm not sure what information about the training script would help. It is a fairly complex script that is training a network for object detection using methods similar to YOLO and SSD. It won't be possible to post the entire set of code, but if there is a way to isolate the source of the delay I may be able to modify it and post that portion.</p>
<p dir="auto">While trying to get insight into the source of the slow-down I also noted that I wasn't able to compile from source using CuDNN4 as noted in the SO posting. This is a secondary issue, but may also warrant investigation.</p>