<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: Custom code included below</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Ubuntu 14.04</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: From pip</li>
<li><strong>TensorFlow version (use command below)</strong>: 1.3.0 and 1.4.0</li>
<li><strong>Python version</strong>: 2.7</li>
<li><strong>Bazel version (if compiling from source)</strong>:</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>:</li>
<li><strong>CUDA/cuDNN version</strong>: cuda-8.0 cudnn-6.0</li>
<li><strong>GPU model and memory</strong>: GTX 1080 8GB</li>
<li><strong>Exact command to reproduce</strong>: python &lt;example_script.py&gt;</li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">Bug. TensorFlow runs out of GPU memory (ResourceExhaustedError) when using version 1.4.0 when running code that runs fine on version 1.3.0. Please see the following script to reproduce.</p>
<h3 dir="auto">Source code / logs</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf
import tensorflow.contrib.slim.nets as nets
import tensorflow.contrib.slim as slim
from tensorflow.contrib.slim.nets import resnet_v2

kBatchSize = 4
kCropSize = 500
kNumClasses = 10

with tf.device('/gpu:0'):
  images = tf.random_normal([kBatchSize, kCropSize, kCropSize, 3])
  labels = tf.constant(0, dtype=tf.int32, shape=[kBatchSize, kCropSize, kCropSize])

  with slim.arg_scope(resnet_v2.resnet_arg_scope()):
    backbone, end_points = resnet_v2.resnet_v2_101(
        images, None, is_training=True, global_pool=False,
        output_stride=8)

    final_conv = tf.layers.conv2d(backbone, kNumClasses, [1, 1], name='final_conv')
    logits = tf.image.resize_bilinear(final_conv, tf.slice(tf.shape(images), [1], [2]))

  loss = tf.nn.sparse_softmax_cross_entropy_with_logits(
        labels=labels, logits=logits)

optimizer = tf.train.GradientDescentOptimizer(learning_rate=.001)
train_op = slim.learning.create_train_op(loss, optimizer)
slim.learning.train(train_op, '/tmp/resnet')"><pre class="notranslate"><code class="notranslate">import tensorflow as tf
import tensorflow.contrib.slim.nets as nets
import tensorflow.contrib.slim as slim
from tensorflow.contrib.slim.nets import resnet_v2

kBatchSize = 4
kCropSize = 500
kNumClasses = 10

with tf.device('/gpu:0'):
  images = tf.random_normal([kBatchSize, kCropSize, kCropSize, 3])
  labels = tf.constant(0, dtype=tf.int32, shape=[kBatchSize, kCropSize, kCropSize])

  with slim.arg_scope(resnet_v2.resnet_arg_scope()):
    backbone, end_points = resnet_v2.resnet_v2_101(
        images, None, is_training=True, global_pool=False,
        output_stride=8)

    final_conv = tf.layers.conv2d(backbone, kNumClasses, [1, 1], name='final_conv')
    logits = tf.image.resize_bilinear(final_conv, tf.slice(tf.shape(images), [1], [2]))

  loss = tf.nn.sparse_softmax_cross_entropy_with_logits(
        labels=labels, logits=logits)

optimizer = tf.train.GradientDescentOptimizer(learning_rate=.001)
train_op = slim.learning.create_train_op(loss, optimizer)
slim.learning.train(train_op, '/tmp/resnet')
</code></pre></div>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: <a href="https://github.com/tkuanlun350/Tensorflow-SegNet">https://github.com/tkuanlun350/Tensorflow-SegNet</a></li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Windows 7 x64</li>
<li><strong>TensorFlow installed from (source or binary)</strong>:  <a href="https://pypi.python.org/pypi/tensorflow-gpu/1.4.0rc1" rel="nofollow">https://pypi.python.org/pypi/tensorflow-gpu/1.4.0rc1</a></li>
<li><strong>TensorFlow version (use command below)</strong>: 1.4.0</li>
<li><strong>Python version</strong>: 3.5</li>
<li><strong>CUDA/cuDNN version</strong>:  Cuda release 8.0, V8.0.60. cuDNN 6.</li>
<li><strong>GPU model and memory</strong>:  NVIDIA P4</li>
<li><strong>Exact command to reproduce</strong>:   <code class="notranslate">c:\python35\python3 main.py --log_dir=./logs --image_dir={image dir} --val_dir= {validation dir} --batch_size=15 --training=True</code></li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">Under 1.3.0 I was able to use a batch size of {15, put your max batch size here} for training.  Under 1.4.0 I get Resource Exhausted errors for that batch size.  So use of GPU resources is going up.  Not the right direction.</p>
<p dir="auto">For me here are the performance effects:</p>
<ul dir="auto">
<li>TensorFlow GPU 1.3.0: 9.8 images/sec for batch size: 15</li>
<li>TensorFlow GPU 1.4.0:  Can't do batch size: 15. 7.8 images/sec for batch size: 12</li>
</ul>
<h3 dir="auto">Source code / logs</h3>
<p dir="auto"><a href="https://github.com/tensorflow/tensorflow/files/1428590/tf_bug2.txt">tf_bug2.txt</a></p>