<h3 dir="auto">System information</h3>
<p dir="auto">OS - High Sierra 10.13<br>
Tensorflow - 1.4<br>
Keras - 2.0.9<br>
CUDA - 9<br>
cuDNN - 7</p>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">CUDA_ERROR_OUT_OF_MEMORY running tensorflow on GPU</p>
<p dir="auto">Simple program:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf
with tf.device('/gpu:0'):
    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
    c = tf.matmul(a, b)

with tf.Session() as sess:
    print (sess.run(c))"><pre class="notranslate"><code class="notranslate">import tensorflow as tf
with tf.device('/gpu:0'):
    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
    c = tf.matmul(a, b)

with tf.Session() as sess:
    print (sess.run(c))
</code></pre></div>
<p dir="auto">Output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="(tensorflow) Smit-Shilu:Desktop smitshilu$ python gputest.py 
2017-11-07 08:55:50.690390: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:856] OS X does not support NUMA - returning NUMA node zero
2017-11-07 08:55:50.690536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1031] Found device 0 with properties: 
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:c3:00.0
totalMemory: 11.00GiB freeMemory: 10.81GiB
2017-11-07 08:55:50.690560: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:0) -&gt; (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:c3:00.0, compute capability: 6.1)
2017-11-07 08:55:50.690914: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 10.27G (11026294784 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2017-11-07 08:55:50.691022: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 9.24G (9923664896 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
[[ 22.  28.]
 [ 49.  64.]]"><pre class="notranslate"><code class="notranslate">(tensorflow) Smit-Shilu:Desktop smitshilu$ python gputest.py 
2017-11-07 08:55:50.690390: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:856] OS X does not support NUMA - returning NUMA node zero
2017-11-07 08:55:50.690536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1031] Found device 0 with properties: 
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:c3:00.0
totalMemory: 11.00GiB freeMemory: 10.81GiB
2017-11-07 08:55:50.690560: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:0) -&gt; (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:c3:00.0, compute capability: 6.1)
2017-11-07 08:55:50.690914: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 10.27G (11026294784 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2017-11-07 08:55:50.691022: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 9.24G (9923664896 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
[[ 22.  28.]
 [ 49.  64.]]
</code></pre></div>
<p dir="auto">CUDA_ERROR_OUT_OF_MEMORY when import Keras after upgrading to 10.13 and 1.4</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; from keras.callbacks import *
Using TensorFlow backend.
2017-11-06 16:30:54.704584: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:856] OS X does not support NUMA - returning NUMA node zero
2017-11-06 16:30:54.704700: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1031] Found device 0 with properties: 
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:c3:00.0
totalMemory: 11.00GiB freeMemory: 10.81GiB
2017-11-06 16:30:54.704725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:0) -&gt; (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:c3:00.0, compute capability: 6.1)
2017-11-06 16:30:54.705019: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 10.27G (11026489344 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2017-11-06 16:30:54.705125: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 9.24G (9923840000 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; from keras.callbacks import *
Using TensorFlow backend.
2017-11-06 16:30:54.704584: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:856] OS X does not support NUMA - returning NUMA node zero
2017-11-06 16:30:54.704700: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1031] Found device 0 with properties: 
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:c3:00.0
totalMemory: 11.00GiB freeMemory: 10.81GiB
2017-11-06 16:30:54.704725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] Creating TensorFlow device (/device:GPU:0) -&gt; (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:c3:00.0, compute capability: 6.1)
2017-11-06 16:30:54.705019: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 10.27G (11026489344 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2017-11-06 16:30:54.705125: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 9.24G (9923840000 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
</code></pre></div>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: <a href="https://github.com/tkuanlun350/Tensorflow-SegNet">https://github.com/tkuanlun350/Tensorflow-SegNet</a></li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Windows 7 x64</li>
<li><strong>TensorFlow installed from (source or binary)</strong>:  <a href="https://pypi.python.org/pypi/tensorflow-gpu/1.4.0rc1" rel="nofollow">https://pypi.python.org/pypi/tensorflow-gpu/1.4.0rc1</a></li>
<li><strong>TensorFlow version (use command below)</strong>: 1.4.0</li>
<li><strong>Python version</strong>: 3.5</li>
<li><strong>CUDA/cuDNN version</strong>:  Cuda release 8.0, V8.0.60. cuDNN 6.</li>
<li><strong>GPU model and memory</strong>:  NVIDIA P4</li>
<li><strong>Exact command to reproduce</strong>:   <code class="notranslate">c:\python35\python3 main.py --log_dir=./logs --image_dir={image dir} --val_dir= {validation dir} --batch_size=15 --training=True</code></li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">Under 1.3.0 I was able to use a batch size of {15, put your max batch size here} for training.  Under 1.4.0 I get Resource Exhausted errors for that batch size.  So use of GPU resources is going up.  Not the right direction.</p>
<p dir="auto">For me here are the performance effects:</p>
<ul dir="auto">
<li>TensorFlow GPU 1.3.0: 9.8 images/sec for batch size: 15</li>
<li>TensorFlow GPU 1.4.0:  Can't do batch size: 15. 7.8 images/sec for batch size: 12</li>
</ul>
<h3 dir="auto">Source code / logs</h3>
<p dir="auto"><a href="https://github.com/tensorflow/tensorflow/files/1428590/tf_bug2.txt">tf_bug2.txt</a></p>