<p dir="auto"><em>Please make sure that this is a bug. As per our <a href="https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md">GitHub Policy</a>, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template</em></p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow):</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Debian</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:</li>
<li>TensorFlow installed from (source or binary):binary</li>
<li>TensorFlow version (use command below):1.13.1,1.14.0，2.0.0a0，1.9.0</li>
<li>Python version:3.7,3.6</li>
<li>Bazel version (if compiling from source):</li>
<li>GCC/Compiler version (if compiling from source):</li>
<li>CUDA/cuDNN version:Cuda 10/7.5.0,7.4.2,7.4.1,7.4.0,Cuda 9 can't support 2060</li>
<li>GPU model and memory:rtx 2060 6GB</li>
</ul>
<p dir="auto">You can collect some of this information using our environment capture<br>
<a href="https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh">script</a><br>
You can also obtain the TensorFlow version with: 1. TF 1.0: <code class="notranslate">python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"</code> 2. TF 2.0: <code class="notranslate">python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"</code></p>
<p dir="auto"><strong>Describe the current behavior</strong></p>
<p dir="auto"><strong>Describe the expected behavior</strong></p>
<p dir="auto"><strong>Code to reproduce the issue</strong><br>
conda activate base<br>
2.0.0-alpha0<br>
2019-05-29 19:58:31.543654: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA<br>
2019-05-29 19:58:31.557759: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1<br>
2019-05-29 19:58:31.670769: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1009] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero<br>
2019-05-29 19:58:31.672348: I tensorflow/compiler/xla/service/service.cc:162] XLA service 0x55c9b57a0a60 executing computations on platform CUDA. Devices:<br>
2019-05-29 19:58:31.672364: I tensorflow/compiler/xla/service/service.cc:169] StreamExecutor device (0): Graphics Device, Compute Capability 7.5<br>
2019-05-29 19:58:31.693201: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2904000000 Hz<br>
2019-05-29 19:58:31.693619: I tensorflow/compiler/xla/service/service.cc:162] XLA service 0x55c9b580db30 executing computations on platform Host. Devices:<br>
2019-05-29 19:58:31.693637: I tensorflow/compiler/xla/service/service.cc:169] StreamExecutor device (0): ,<br>
2019-05-29 19:58:31.693791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1467] Found device 0 with properties:<br>
name: Graphics Device major: 7 minor: 5 memoryClockRate(GHz): 1.71<br>
pciBusID: 0000:01:00.0<br>
totalMemory: 5.76GiB freeMemory: 5.17GiB<br>
2019-05-29 19:58:31.693804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1546] Adding visible gpu devices: 0<br>
2019-05-29 19:58:31.693844: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0<br>
2019-05-29 19:58:31.694402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1015] Device interconnect StreamExecutor with strength 1 edge matrix:<br>
2019-05-29 19:58:31.694413: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1021] 0<br>
2019-05-29 19:58:31.694419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1034] 0: N<br>
2019-05-29 19:58:31.694516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1149] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4990 MB memory) -&gt; physical GPU (device: 0, name: Graphics Device, pci bus id: 0000:01:00.0, compute capability: 7.5)<br>
Number of training examples: 60000<br>
Number of test examples: 10000<br>
Epoch 1/5<br>
2019-05-29 19:58:32.786674: W ./tensorflow/core/framework/model.h:202] Encountered a stop event that was not preceded by a start event.<br>
2019-05-29 19:58:36.114172: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0<br>
2019-05-29 19:58:36.288348: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7<br>
2019-05-29 19:58:36.934479: E tensorflow/stream_executor/cuda/cuda_dnn.cc:338] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR<br>
2019-05-29 19:58:36.945020: E tensorflow/stream_executor/cuda/cuda_dnn.cc:338] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR<br>
2019-05-29 19:58:36.945127: W tensorflow/core/common_runtime/base_collective_executor.cc:214] BaseCollectiveExecutor::StartAbort Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.<br>
[[{{node conv2d/Conv2D}}]]</p>
<p dir="auto"><strong>Other info / logs</strong><br>
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.</p>
<p dir="auto"><em>Please make sure that this is a bug. As per our <a href="https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md">GitHub Policy</a>, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template</em></p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes and No (described below)</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Manjaro</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:</li>
<li>TensorFlow installed from (source or binary): tf-nightly-gpu (Dec 19, r1.13)</li>
<li>TensorFlow version (use command below): 1.13.0-dev20181219</li>
<li>Python version: 3.7.1</li>
<li>Bazel version (if compiling from source):</li>
<li>GCC/Compiler version (if compiling from source):</li>
<li>CUDA/cuDNN version: CUDA 10 with cuDNN 7.4.1</li>
<li>GPU model and memory: RTX 2070 8GB</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong><br>
I'm running the CNN model on MNIST. When I'm running with the GPU, I am encountering<br>
<code class="notranslate">2018-12-20 20:09:13.644176: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR</code></p>
<p dir="auto">I did some digging and realized that it is a memory issue (which shouldn't be the case as I have 32GB of RAM and 64GB of swap. I ran htop when running the model and I have 20+GB free, which is more than enough to fit the 8GB vRAM mappings.</p>
<p dir="auto">Using the <code class="notranslate">gpu_options.allow_growth = True</code> gets the model to work properly, and setting <code class="notranslate">os.environ['CUDA_VISIBLE_DEVICES'] = '-1'</code> also works. This means that I AM facing a memory issue, but I don't see how.</p>
<p dir="auto">Also, using <code class="notranslate">gpu_options.allow_growth = True</code> does not fix the same issue when trying to run tensorflow/models/official/mnist/ model, which should have a similar behavior with my code.</p>
<p dir="auto"><strong>Code to reproduce the issue</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import os
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
import math
import time
# Killing optional CPU driver warnings
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
# os.environ['CUDA_VISIBLE_DEVICES'] = '-1'
tf.logging.set_verbosity(tf.logging.ERROR)


class Model:

    def __init__(self, image, label):
        &quot;&quot;&quot;
        A Model class contains a computational graph that classifies images
        to predictions. Each of its methods builds part of the graph
        on Model initialization. Do not modify the constructor, as doing so
        would break the autograder. You may, however, add class variables
        to use in your graph-building. e.g. learning rate, 

        image: the input image to the computational graph as a tensor
        label: the correct label of an image as a tensor
        prediction: the output prediction of the computational graph,
                    produced by self.forward_pass()
        optimize: the model's optimizing tensor produced by self.optimizer()
        loss: the model's loss produced by computing self.loss_function()
        accuracy: the model's prediction accuracy
        &quot;&quot;&quot;
        self.image = image
        self.label = label

        # TO-DO: Add any class variables you want to use.

        self.prediction = self.forward_pass()
        self.loss = self.loss_function()
        self.optimize = self.optimizer()
        self.accuracy = self.accuracy_function()

    def forward_pass(self):
        &quot;&quot;&quot;
        Predicts a label given an image using convolution layers

        :return: the prediction as a tensor
        &quot;&quot;&quot;
        filter_1 = tf.Variable(tf.truncated_normal([3, 3, 1, 8], stddev=0.1))
        conv_1 = tf.nn.conv2d(self.image, filter_1, [1, 1, 1, 1], &quot;SAME&quot;)

        reshaped = tf.reshape(conv_1, shape=[50, -1])

        L1 = reshaped.shape[1].value
        L2 = 500
        W1 = tf.Variable(tf.random_normal([L1, L2], mean=0, stddev=0.01))
        b1 = tf.Variable(tf.random_normal([L2], mean=0, stddev=0.01))
        relu_1 = tf.nn.relu(tf.matmul(reshaped, W1) + b1)

        W2 = tf.Variable(tf.random_normal([L2, 10], mean=0, stddev=0.01))
        b2 = tf.Variable(tf.random_normal([10], mean=0, stddev=0.01))
        logits = tf.nn.relu(tf.matmul(relu_1, W2) + b2)
        return logits

    def loss_function(self):
        &quot;&quot;&quot;
        Calculates the model cross-entropy loss

        :return: the loss of the model as a tensor
        &quot;&quot;&quot;
        loss = tf.losses.softmax_cross_entropy(onehot_labels=self.label, logits=self.prediction)
        return loss

    def optimizer(self):
        &quot;&quot;&quot;
        Optimizes the model loss using an Adam Optimizer

        :return: the optimizer as a tensor
        &quot;&quot;&quot;
        learning_rate = 0.1
        sgd = tf.train.GradientDescentOptimizer(learning_rate)
        train = sgd.minimize(self.loss)
        return train

    def accuracy_function(self):
        &quot;&quot;&quot;
        Calculates the model's prediction accuracy by comparing
        predictions to correct labels – no need to modify this

        :return: the accuracy of the model as a tensor
        &quot;&quot;&quot;
        correct_prediction = tf.equal(tf.argmax(self.prediction, 1),
                                      tf.argmax(self.label, 1))
        return tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


def main():
    t_start = time.time()

    mnist = input_data.read_data_sets(&quot;data/mnist/&quot;, one_hot=True)
    batch_sz = 50
    batch = 2000

    inputs = tf.placeholder(shape=[batch_sz, 28, 28, 1], dtype=tf.float32)
    labels = tf.placeholder(shape=[batch_sz, 10], dtype=tf.float32)

    model = Model(inputs, labels)

    session_config = tf.ConfigProto(gpu_options=tf.GPUOptions(allow_growth=True))
    sess = tf.Session(config=session_config)

    # sess = tf.Session()

    sess.run(tf.global_variables_initializer())
    for i in range(batch):
        next_image, next_label = mnist.train.next_batch(batch_sz)
        next_image = next_image.reshape((batch_sz, 28, 28, 1))
        sess.run(model.optimize, feed_dict={inputs: next_image, labels: next_label})

    acc, test_images, test_labels = 0, mnist.test.images, mnist.test.labels
    test_batch = math.ceil(len(test_images) / batch_sz)
    for i in range(test_batch):
        batch_images = test_images[i * batch_sz: (i + 1) * batch_sz]
        batch_images = batch_images.reshape((batch_sz, 28, 28, 1))
        batch_labes = test_labels[i * batch_sz: (i + 1) * batch_sz]
        acc += sess.run(model.accuracy, feed_dict={inputs: batch_images, labels: batch_labes})
    acc /= test_batch
    print(acc)

    print(time.time() - t_start, 'seconds')

    return


if __name__ == '__main__':
    main()"><pre class="notranslate"><code class="notranslate">import os
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
import math
import time
# Killing optional CPU driver warnings
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
# os.environ['CUDA_VISIBLE_DEVICES'] = '-1'
tf.logging.set_verbosity(tf.logging.ERROR)


class Model:

    def __init__(self, image, label):
        """
        A Model class contains a computational graph that classifies images
        to predictions. Each of its methods builds part of the graph
        on Model initialization. Do not modify the constructor, as doing so
        would break the autograder. You may, however, add class variables
        to use in your graph-building. e.g. learning rate, 

        image: the input image to the computational graph as a tensor
        label: the correct label of an image as a tensor
        prediction: the output prediction of the computational graph,
                    produced by self.forward_pass()
        optimize: the model's optimizing tensor produced by self.optimizer()
        loss: the model's loss produced by computing self.loss_function()
        accuracy: the model's prediction accuracy
        """
        self.image = image
        self.label = label

        # TO-DO: Add any class variables you want to use.

        self.prediction = self.forward_pass()
        self.loss = self.loss_function()
        self.optimize = self.optimizer()
        self.accuracy = self.accuracy_function()

    def forward_pass(self):
        """
        Predicts a label given an image using convolution layers

        :return: the prediction as a tensor
        """
        filter_1 = tf.Variable(tf.truncated_normal([3, 3, 1, 8], stddev=0.1))
        conv_1 = tf.nn.conv2d(self.image, filter_1, [1, 1, 1, 1], "SAME")

        reshaped = tf.reshape(conv_1, shape=[50, -1])

        L1 = reshaped.shape[1].value
        L2 = 500
        W1 = tf.Variable(tf.random_normal([L1, L2], mean=0, stddev=0.01))
        b1 = tf.Variable(tf.random_normal([L2], mean=0, stddev=0.01))
        relu_1 = tf.nn.relu(tf.matmul(reshaped, W1) + b1)

        W2 = tf.Variable(tf.random_normal([L2, 10], mean=0, stddev=0.01))
        b2 = tf.Variable(tf.random_normal([10], mean=0, stddev=0.01))
        logits = tf.nn.relu(tf.matmul(relu_1, W2) + b2)
        return logits

    def loss_function(self):
        """
        Calculates the model cross-entropy loss

        :return: the loss of the model as a tensor
        """
        loss = tf.losses.softmax_cross_entropy(onehot_labels=self.label, logits=self.prediction)
        return loss

    def optimizer(self):
        """
        Optimizes the model loss using an Adam Optimizer

        :return: the optimizer as a tensor
        """
        learning_rate = 0.1
        sgd = tf.train.GradientDescentOptimizer(learning_rate)
        train = sgd.minimize(self.loss)
        return train

    def accuracy_function(self):
        """
        Calculates the model's prediction accuracy by comparing
        predictions to correct labels – no need to modify this

        :return: the accuracy of the model as a tensor
        """
        correct_prediction = tf.equal(tf.argmax(self.prediction, 1),
                                      tf.argmax(self.label, 1))
        return tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


def main():
    t_start = time.time()

    mnist = input_data.read_data_sets("data/mnist/", one_hot=True)
    batch_sz = 50
    batch = 2000

    inputs = tf.placeholder(shape=[batch_sz, 28, 28, 1], dtype=tf.float32)
    labels = tf.placeholder(shape=[batch_sz, 10], dtype=tf.float32)

    model = Model(inputs, labels)

    session_config = tf.ConfigProto(gpu_options=tf.GPUOptions(allow_growth=True))
    sess = tf.Session(config=session_config)

    # sess = tf.Session()

    sess.run(tf.global_variables_initializer())
    for i in range(batch):
        next_image, next_label = mnist.train.next_batch(batch_sz)
        next_image = next_image.reshape((batch_sz, 28, 28, 1))
        sess.run(model.optimize, feed_dict={inputs: next_image, labels: next_label})

    acc, test_images, test_labels = 0, mnist.test.images, mnist.test.labels
    test_batch = math.ceil(len(test_images) / batch_sz)
    for i in range(test_batch):
        batch_images = test_images[i * batch_sz: (i + 1) * batch_sz]
        batch_images = batch_images.reshape((batch_sz, 28, 28, 1))
        batch_labes = test_labels[i * batch_sz: (i + 1) * batch_sz]
        acc += sess.run(model.accuracy, feed_dict={inputs: batch_images, labels: batch_labes})
    acc /= test_batch
    print(acc)

    print(time.time() - t_start, 'seconds')

    return


if __name__ == '__main__':
    main()
</code></pre></div>