<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: Yes</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Ubuntu 16.04</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: Source</li>
<li><strong>TensorFlow version (use command below)</strong>: r1.5</li>
<li><strong>Python version</strong>: 3.5.2</li>
<li><strong>Bazel version (if compiling from source)</strong>: 0.9.0</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>: 5.4.0</li>
<li><strong>CUDA/cuDNN version</strong>: Cuda 8.0 and cuDNN 7.0</li>
<li><strong>GPU model and memory</strong>: Jetson TX2</li>
</ul>
<h3 dir="auto">The problem</h3>
<p dir="auto">I created a fully convolutional float 16 (half precision) neural network in tensorflow. When I run this network with some inputs, the inference time is slower than when I run the same network in float 32 (full precision) mode.<br>
I should also note that the following variables are set:</p>
<p dir="auto"><code class="notranslate">os.environ['TF_FP16_CONV_USE_FP32_COMPUTE'] = '0' os.environ['TF_FP16_MATMUL_USE_FP32_COMPUTE'] = '0'</code></p>
<p dir="auto">As Nvidia Jetson TX2 support FP16 operations, I expected an inference time not worse than when I use FP32, but surprisingly it is about 1.5 times worse! (36 miliseconds vs 22 miliseconds). I guess it is becuase of the overhead of internal type conversion in the tensorflow core between float16 and float32!</p>
<p dir="auto">Is it a problem with Tensorflow or TX2?</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="136722135" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/1300" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/1300/hovercard" href="https://github.com/tensorflow/tensorflow/issues/1300">#1300</a></p>
<p dir="auto">I have got a nvidia p100 GPU which is support fp16, and I run the TF case 'cifar10_train.py'. Without option '--use_fp16', the performance is also 1600 examples/sec, and with the option '--use_fp16', the performance down to 500 examples/sec. Any ideas about this issue?</p>
<p dir="auto">userid@ubuntu-WK-4xP100:<del>/weike/tensorflow-r0.11/tensorflow/models/image/cifar10$ vi cifar10_train.py<br>
userid@ubuntu-WK-4xP100:</del>/weike/tensorflow-r0.11/tensorflow/models/image/cifar10$ python cifar10_train.py<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so.8.0 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so.5 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so.8.0 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so.8.0 locally</p>
<blockquote>
<blockquote>
<p dir="auto">Downloading cifar-10-binary.tar.gz 100.0%<br>
...<br>
2016-11-14 00:07:57.143739: step 0, loss = 4.67 (12.1 examples/sec; 10.549 sec/batch)<br>
2016-11-14 00:07:58.395209: step 10, loss = 4.57 (1693.6 examples/sec; 0.076 sec/batch)<br>
2016-11-14 00:07:59.177525: step 20, loss = 4.97 (1668.9 examples/sec; 0.077 sec/batch)<br>
2016-11-14 00:07:59.957789: step 30, loss = 4.43 (1588.3 examples/sec; 0.081 sec/batch)<br>
2016-11-14 00:08:00.738431: step 40, loss = 4.52 (1690.5 examples/sec; 0.076 sec/batch)<br>
2016-11-14 00:08:01.501940: step 50, loss = 4.33 (1680.4 examples/sec; 0.076 sec/batch)<br>
2016-11-14 00:08:02.241604: step 60, loss = 4.20 (1733.4 examples/sec; 0.074 sec/batch)<br>
2016-11-14 00:08:03.001845: step 70, loss = 4.27 (1706.0 examples/sec; 0.075 sec/batch)<br>
2016-11-14 00:08:03.765522: step 80, loss = 4.18 (1601.3 examples/sec; 0.080 sec/batch)<br>
2016-11-14 00:08:04.516780: step 90, loss = 4.25 (1646.3 examples/sec; 0.078 sec/batch)</p>
</blockquote>
</blockquote>
<p dir="auto">userid@ubuntu-WK-4xP100:~/weike/tensorflow-r0.11/tensorflow/models/image/cifar10$ python cifar10_train.py --use_fp16<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so.8.0 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so.5 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so.8.0 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally<br>
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so.8.0 locally<br>
Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.<br>
..<br>
2016-11-14 00:09:09.382854: step 0, loss = 4.67 (12.3 examples/sec; 10.411 sec/batch)<br>
2016-11-14 00:09:11.923842: step 10, loss = 4.58 (659.6 examples/sec; 0.194 sec/batch)<br>
2016-11-14 00:09:13.918448: step 20, loss = 6.62 (460.6 examples/sec; 0.278 sec/batch)<br>
2016-11-14 00:09:16.211809: step 30, loss = 4.37 (583.7 examples/sec; 0.219 sec/batch)<br>
2016-11-14 00:09:18.327690: step 40, loss = 4.30 (618.3 examples/sec; 0.207 sec/batch)<br>
2016-11-14 00:09:20.395409: step 50, loss = 4.37 (643.8 examples/sec; 0.199 sec/batch)<br>
2016-11-14 00:09:22.466230: step 60, loss = 4.32 (574.0 examples/sec; 0.223 sec/batch)<br>
2016-11-14 00:09:24.533225: step 70, loss = 4.17 (646.2 examples/sec; 0.198 sec/batch)<br>
2016-11-14 00:09:26.609277: step 80, loss = 2.59 (601.5 examples/sec; 0.213 sec/batch)<br>
2016-11-14 00:09:28.703648: step 90, loss = 4.15 (625.5 examples/sec; 0.205 sec/batch)</p>