<p dir="auto"><em>Please make sure that this is a build/installation issue. As per our <a href="https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md">GitHub Policy</a>, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template</em></p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>OS Platform and Distribution: Ubuntu 16.04<br>
Using Docker (latest-gpu image)</li>
<li>Mobile Device: No</li>
<li>TensorFlow installed from (source or binary): N/A</li>
<li>TensorFlow version: 1.13.0-rc1</li>
<li>Python version:  Python 3.5.2</li>
<li>Installed using virtualenv? pip? conda?: N/A</li>
<li>Bazel version (if compiling from source): N/A</li>
<li>GCC/Compiler version (if compiling from source): N/A</li>
<li>CUDA/cuDNN version:<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2018 NVIDIA Corporation<br>
Built on Sat_Aug_25_21:08:01_CDT_2018<br>
Cuda compilation tools, release 10.0, V10.0.130</li>
<li>GPU model and memory: Quadro M1200</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="==============NVSMI LOG==============

Timestamp                           : Mon Feb 25 00:50:20 2019
Driver Version                      : 410.78
CUDA Version                        : 10.0

Attached GPUs                       : 1
GPU 00000000:01:00.0
    Product Name                    : Quadro M1200
    Product Brand                   : Quadro
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Enabled
    Accounting Mode                 : Disabled
    Accounting Mode Buffer Size     : 4000
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : N/A
    GPU UUID                        : GPU-d9093d17-7927-a053-9104-426e68b1d4ac
    Minor Number                    : 0
    VBIOS Version                   : 82.07.BB.00.13
    MultiGPU Board                  : No
    Board ID                        : 0x100
    GPU Part Number                 : N/A
    Inforom Version
        Image Version               : N/A
        OEM Object                  : N/A
        ECC Object                  : N/A
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization mode         : None
    IBMNPU
        Relaxed Ordering Mode       : N/A
    PCI
        Bus                         : 0x01
        Device                      : 0x00
        Domain                      : 0x0000
        Device Id                   : 0x13B610DE
        Bus Id                      : 00000000:01:00.0
        Sub System Id               : 0x224D17AA
        GPU Link Info
            PCIe Generation
                Max                 : 3
                Current             : 3
            Link Width
                Max                 : 16x
                Current             : 16x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays since reset         : 0
        Tx Throughput               : 0 KB/s
        Rx Throughput               : 0 KB/s
    Fan Speed                       : N/A
    Performance State               : P0
    Clocks Throttle Reasons
        Idle                        : Not Active
        Applications Clocks Setting : Active
        SW Power Cap                : Not Active
        HW Slowdown                 : Not Active
            HW Thermal Slowdown     : N/A
            HW Power Brake Slowdown : N/A
        Sync Boost                  : Not Active
        SW Thermal Slowdown         : Not Active
        Display Clock Setting       : Not Active
    FB Memory Usage
        Total                       : 4043 MiB
        Used                        : 3813 MiB
        Free                        : 230 MiB
    BAR1 Memory Usage
        Total                       : 256 MiB
        Used                        : 3 MiB
        Free                        : 253 MiB
    Compute Mode                    : Default
    Utilization
        Gpu                         : 0 %
        Memory                      : 0 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    Encoder Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    FBC Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    Ecc Mode
        Current                     : N/A
        Pending                     : N/A
    ECC Errors
        Volatile
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
        Aggregate
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
    Retired Pages
        Single Bit ECC              : N/A
        Double Bit ECC              : N/A
        Pending                     : N/A
    Temperature
        GPU Current Temp            : 37 C
        GPU Shutdown Temp           : N/A
        GPU Slowdown Temp           : 96 C
        GPU Max Operating Temp      : 92 C
        Memory Current Temp         : N/A
        Memory Max Operating Temp   : N/A
    Power Readings
        Power Management            : N/A
        Power Draw                  : N/A
        Power Limit                 : N/A
        Default Power Limit         : N/A
        Enforced Power Limit        : N/A
        Min Power Limit             : N/A
        Max Power Limit             : N/A
    Clocks
        Graphics                    : 993 MHz
        SM                          : 993 MHz
        Memory                      : 2505 MHz
        Video                       : 893 MHz
    Applications Clocks
        Graphics                    : N/A
        Memory                      : N/A
    Default Applications Clocks
        Graphics                    : N/A
        Memory                      : N/A
    Max Clocks
        Graphics                    : 1150 MHz
        SM                          : 1150 MHz
        Memory                      : 2505 MHz
        Video                       : 1035 MHz
    Max Customer Boost Clocks
        Graphics                    : N/A
    Clock Policy
        Auto Boost                  : N/A
        Auto Boost Default          : N/A
    Processes
        Process ID                  : 1123
            Type                    : G
            Name                    : /usr/lib/xorg/Xorg
            Used GPU Memory         : 8 MiB
        Process ID                  : 31763
            Type                    : C
            Name                    : python
            Used GPU Memory         : 3791 MiB"><pre class="notranslate"><code class="notranslate">==============NVSMI LOG==============

Timestamp                           : Mon Feb 25 00:50:20 2019
Driver Version                      : 410.78
CUDA Version                        : 10.0

Attached GPUs                       : 1
GPU 00000000:01:00.0
    Product Name                    : Quadro M1200
    Product Brand                   : Quadro
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Enabled
    Accounting Mode                 : Disabled
    Accounting Mode Buffer Size     : 4000
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : N/A
    GPU UUID                        : GPU-d9093d17-7927-a053-9104-426e68b1d4ac
    Minor Number                    : 0
    VBIOS Version                   : 82.07.BB.00.13
    MultiGPU Board                  : No
    Board ID                        : 0x100
    GPU Part Number                 : N/A
    Inforom Version
        Image Version               : N/A
        OEM Object                  : N/A
        ECC Object                  : N/A
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization mode         : None
    IBMNPU
        Relaxed Ordering Mode       : N/A
    PCI
        Bus                         : 0x01
        Device                      : 0x00
        Domain                      : 0x0000
        Device Id                   : 0x13B610DE
        Bus Id                      : 00000000:01:00.0
        Sub System Id               : 0x224D17AA
        GPU Link Info
            PCIe Generation
                Max                 : 3
                Current             : 3
            Link Width
                Max                 : 16x
                Current             : 16x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays since reset         : 0
        Tx Throughput               : 0 KB/s
        Rx Throughput               : 0 KB/s
    Fan Speed                       : N/A
    Performance State               : P0
    Clocks Throttle Reasons
        Idle                        : Not Active
        Applications Clocks Setting : Active
        SW Power Cap                : Not Active
        HW Slowdown                 : Not Active
            HW Thermal Slowdown     : N/A
            HW Power Brake Slowdown : N/A
        Sync Boost                  : Not Active
        SW Thermal Slowdown         : Not Active
        Display Clock Setting       : Not Active
    FB Memory Usage
        Total                       : 4043 MiB
        Used                        : 3813 MiB
        Free                        : 230 MiB
    BAR1 Memory Usage
        Total                       : 256 MiB
        Used                        : 3 MiB
        Free                        : 253 MiB
    Compute Mode                    : Default
    Utilization
        Gpu                         : 0 %
        Memory                      : 0 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    Encoder Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    FBC Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    Ecc Mode
        Current                     : N/A
        Pending                     : N/A
    ECC Errors
        Volatile
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
        Aggregate
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
    Retired Pages
        Single Bit ECC              : N/A
        Double Bit ECC              : N/A
        Pending                     : N/A
    Temperature
        GPU Current Temp            : 37 C
        GPU Shutdown Temp           : N/A
        GPU Slowdown Temp           : 96 C
        GPU Max Operating Temp      : 92 C
        Memory Current Temp         : N/A
        Memory Max Operating Temp   : N/A
    Power Readings
        Power Management            : N/A
        Power Draw                  : N/A
        Power Limit                 : N/A
        Default Power Limit         : N/A
        Enforced Power Limit        : N/A
        Min Power Limit             : N/A
        Max Power Limit             : N/A
    Clocks
        Graphics                    : 993 MHz
        SM                          : 993 MHz
        Memory                      : 2505 MHz
        Video                       : 893 MHz
    Applications Clocks
        Graphics                    : N/A
        Memory                      : N/A
    Default Applications Clocks
        Graphics                    : N/A
        Memory                      : N/A
    Max Clocks
        Graphics                    : 1150 MHz
        SM                          : 1150 MHz
        Memory                      : 2505 MHz
        Video                       : 1035 MHz
    Max Customer Boost Clocks
        Graphics                    : N/A
    Clock Policy
        Auto Boost                  : N/A
        Auto Boost Default          : N/A
    Processes
        Process ID                  : 1123
            Type                    : G
            Name                    : /usr/lib/xorg/Xorg
            Used GPU Memory         : 8 MiB
        Process ID                  : 31763
            Type                    : C
            Name                    : python
            Used GPU Memory         : 3791 MiB
</code></pre></div>
<p dir="auto"><strong>Describe the problem</strong></p>
<p dir="auto">When using my GPU, it takes several minutes (just over 4 minutes) to initialise to do anything. Issue does not exist when using CPU</p>
<p dir="auto"><strong>Provide the exact sequence of commands / steps that you executed before running into the problem</strong></p>
<p dir="auto"><code class="notranslate">docker run -it -u $(id -u):$(id -g) --runtime=nvidia -v $(realpath ~/tensorflow):/tf/tensorflow tensorflow/tensorflow:latest-gpu bash</code></p>
<p dir="auto"><code class="notranslate">python test.py</code></p>
<p dir="auto">contents of test.py:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf
mnist = tf.keras.datasets.mnist

(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(512, activation=tf.nn.relu),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)"><pre class="notranslate"><code class="notranslate">import tensorflow as tf
mnist = tf.keras.datasets.mnist

(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(512, activation=tf.nn.relu),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)
</code></pre></div>
<p dir="auto"><strong>Any other info / logs</strong><br>
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.</p>
<p dir="auto">logs while running test script</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step
11501568/11490434 [==============================] - 0s 0us/step
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
2019-02-25 05:46:52.561440: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-02-25 05:46:52.628689: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-25 05:46:52.629997: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x50be7d0 executing computations on platform CUDA. Devices:
2019-02-25 05:46:52.630035: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): Quadro M1200, Compute Capability 5.0
2019-02-25 05:46:52.664820: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2808000000 Hz
2019-02-25 05:46:52.666234: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5128500 executing computations on platform Host. Devices:
2019-02-25 05:46:52.666318: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): &lt;undefined&gt;, &lt;undefined&gt;
2019-02-25 05:46:52.666979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: 
name: Quadro M1200 major: 5 minor: 0 memoryClockRate(GHz): 1.148
pciBusID: 0000:01:00.0
totalMemory: 3.95GiB freeMemory: 3.90GiB
2019-02-25 05:46:52.667052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-02-25 05:46:52.669065: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-25 05:46:52.669122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 
2019-02-25 05:46:52.669152: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N 
2019-02-25 05:46:52.669563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3696 MB memory) -&gt; physical GPU (device: 0, name: Quadro M1200, pci bus id: 0000:01:00.0, compute capability: 5.0)
Epoch 1/5
2019-02-25 05:51:01.254939: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
60000/60000 [==============================] - 5s 84us/sample - loss: 0.2207 - acc: 0.9348
Epoch 2/5
60000/60000 [==============================] - 5s 79us/sample - loss: 0.0960 - acc: 0.9714
Epoch 3/5
60000/60000 [==============================] - 5s 78us/sample - loss: 0.0697 - acc: 0.9774
Epoch 4/5
60000/60000 [==============================] - 5s 79us/sample - loss: 0.0536 - acc: 0.9826
Epoch 5/5
60000/60000 [==============================] - 5s 76us/sample - loss: 0.0430 - acc: 0.9857
10000/10000 [==============================] - 0s 29us/sample - loss: 0.0606 - acc: 0.9813"><pre class="notranslate"><code class="notranslate">Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step
11501568/11490434 [==============================] - 0s 0us/step
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
2019-02-25 05:46:52.561440: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-02-25 05:46:52.628689: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-25 05:46:52.629997: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x50be7d0 executing computations on platform CUDA. Devices:
2019-02-25 05:46:52.630035: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): Quadro M1200, Compute Capability 5.0
2019-02-25 05:46:52.664820: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2808000000 Hz
2019-02-25 05:46:52.666234: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5128500 executing computations on platform Host. Devices:
2019-02-25 05:46:52.666318: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): &lt;undefined&gt;, &lt;undefined&gt;
2019-02-25 05:46:52.666979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: 
name: Quadro M1200 major: 5 minor: 0 memoryClockRate(GHz): 1.148
pciBusID: 0000:01:00.0
totalMemory: 3.95GiB freeMemory: 3.90GiB
2019-02-25 05:46:52.667052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-02-25 05:46:52.669065: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-25 05:46:52.669122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 
2019-02-25 05:46:52.669152: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N 
2019-02-25 05:46:52.669563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3696 MB memory) -&gt; physical GPU (device: 0, name: Quadro M1200, pci bus id: 0000:01:00.0, compute capability: 5.0)
Epoch 1/5
2019-02-25 05:51:01.254939: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
60000/60000 [==============================] - 5s 84us/sample - loss: 0.2207 - acc: 0.9348
Epoch 2/5
60000/60000 [==============================] - 5s 79us/sample - loss: 0.0960 - acc: 0.9714
Epoch 3/5
60000/60000 [==============================] - 5s 78us/sample - loss: 0.0697 - acc: 0.9774
Epoch 4/5
60000/60000 [==============================] - 5s 79us/sample - loss: 0.0536 - acc: 0.9826
Epoch 5/5
60000/60000 [==============================] - 5s 76us/sample - loss: 0.0430 - acc: 0.9857
10000/10000 [==============================] - 0s 29us/sample - loss: 0.0606 - acc: 0.9813
</code></pre></div>
<p dir="auto">I noticed that tensorflow always takes about ~2min before it actually starts to compute. I've been trying to find out, why this happens, and nothing really worked so far.</p>
<p dir="auto"><a href="https://www.tensorflow.org/install/install_windows" rel="nofollow">Tensorflow site</a> says, I should use CUDA® Toolkit 9.0 and cuDNN v7.0. I have CUDA 9.0, so I downloaded CuDNN 7.0.5 for CUDA 9.0 and pasted the files to *C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0*, overwriting the ones form cuDNN 7.1.2, which I tested earlier. To make sure, I pip-installed tensorflow-gpu into a fresh anaconda env. See install <a href="https://pastebin.com/rjiV1s3b" rel="nofollow">here</a>. The issue is still the same.</p>
<p dir="auto">CUDA works, since it prints the  <em>'Hello, TensorFlow!'</em>, when I use the official test example, but before that it takes like 2minutes every time!</p>
<p dir="auto">When I tested this with <a href="https://drive.google.com/drive/folders/1lVK_ABvVHzVYKs7X5SUhcZFBgKpC41Qw" rel="nofollow">another wheel</a> (<a href="http://www.python36.com/install-tensorflow-gpu-windows/" rel="nofollow">which is linked in this tutorial</a>, I did not compile it myself.) on cuda 9.1/cudnn7.0.5, I had the same issues. A NVIDIA employee <a href="https://stackoverflow.com/questions/49770217/why-does-cuda-initialisation-take-so-long-python-vscode-anaconda-tensorflow" rel="nofollow">on stackoverflow</a> suggested, I may be hitting a lengthy JIT compile step, because the GTX 1080 has compute capability of 6.1, which the wheel I used may not be compiled for.</p>
<p dir="auto">So I tried to find wheels for tensorflow with compute capability 6.1 for windows, but <a href="https://github.com/fo40225/tensorflow-windows-wheel/tree/master/1.5.0/py36/GPU/cuda91cudnn7avx2">the only one I found</a> and tested produced the same problem.</p>
<p dir="auto">Am I doing something wrong here, or do I just have to accept the 2min delay everytime I start my tensorflow/keras scripts?</p>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>:<br>
Code:</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import time
start_time = time.time()
import tensorflow as tf
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
print(sess.run(c))
timer = time.time()
print(timer - start_time)"><pre class="notranslate"><code class="notranslate">import time
start_time = time.time()
import tensorflow as tf
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
print(sess.run(c))
timer = time.time()
print(timer - start_time)
</code></pre></div>
<p dir="auto">Output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="(tf_clean) C:\python_code\test&gt;C:/anaconda/envs/tf_clean/python.exe c:/python_code/test/tf_test.py
2018-04-18 14:36:04.376661: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this
TensorFlow binary was not compiled to use: AVX2
2018-04-18 14:36:04.689661: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1344] Found device 0 with properties:
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7335
pciBusID: 0000:01:00.0
totalMemory: 8.00GiB freeMemory: 6.60GiB
2018-04-18 14:36:04.699485: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1423] Adding visible gpu devices: 0
2018-04-18 14:38:12.227561: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-04-18 14:38:12.234504: I T:\src\github\tens2018-04-18 14:38:12.237156: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:930] 0:   N
2018-04-18 14:38:12.240997: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6379 MB memory) -&gt; physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -&gt; device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1
2018-04-18 14:38:12.548288: I T:\src\github\tensorflow\tensorflow\core\common_runtime\direct_session.cc:297] Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -&gt; device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1
MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0
2018-04-18 14:38:12.559262: I T:\src\github\tensorflow\tensorflow\core\common_runtime\placer.cc:884] MatMul: (MatMul)/job:localhost/replica:0/task:0/device:GPU:0
b: (Const): /job:localhost/replica:0/task:0/device:GPU:0
2018-04-18 14:38:12.564847: I T:\src\github\tensorflow\tensorflow\core\common_runtime\placer.cc:884] b: (Const)/job:localhost/replica:0/task:0/device:GPU:0
a: (Const): /job:localhost/replica:0/task:0/device:GPU:0
2018-04-18 14:38:12.570545: I T:\src\github\tensorflow\tensorflow\core\common_runtime\placer.cc:884] a: (Const)/job:localhost/replica:0/task:0/device:GPU:0
[[22. 28.]
 [49. 64.]]
129.14624643325806"><pre class="notranslate"><code class="notranslate">(tf_clean) C:\python_code\test&gt;C:/anaconda/envs/tf_clean/python.exe c:/python_code/test/tf_test.py
2018-04-18 14:36:04.376661: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this
TensorFlow binary was not compiled to use: AVX2
2018-04-18 14:36:04.689661: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1344] Found device 0 with properties:
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7335
pciBusID: 0000:01:00.0
totalMemory: 8.00GiB freeMemory: 6.60GiB
2018-04-18 14:36:04.699485: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1423] Adding visible gpu devices: 0
2018-04-18 14:38:12.227561: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-04-18 14:38:12.234504: I T:\src\github\tens2018-04-18 14:38:12.237156: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:930] 0:   N
2018-04-18 14:38:12.240997: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6379 MB memory) -&gt; physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -&gt; device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1
2018-04-18 14:38:12.548288: I T:\src\github\tensorflow\tensorflow\core\common_runtime\direct_session.cc:297] Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -&gt; device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1
MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0
2018-04-18 14:38:12.559262: I T:\src\github\tensorflow\tensorflow\core\common_runtime\placer.cc:884] MatMul: (MatMul)/job:localhost/replica:0/task:0/device:GPU:0
b: (Const): /job:localhost/replica:0/task:0/device:GPU:0
2018-04-18 14:38:12.564847: I T:\src\github\tensorflow\tensorflow\core\common_runtime\placer.cc:884] b: (Const)/job:localhost/replica:0/task:0/device:GPU:0
a: (Const): /job:localhost/replica:0/task:0/device:GPU:0
2018-04-18 14:38:12.570545: I T:\src\github\tensorflow\tensorflow\core\common_runtime\placer.cc:884] a: (Const)/job:localhost/replica:0/task:0/device:GPU:0
[[22. 28.]
 [49. 64.]]
129.14624643325806
</code></pre></div>
<ul dir="auto">
<li>
<p dir="auto"><strong>OS Platform and Distribution</strong>:<br>
Windows 10 Education (Version 10.0.16299 Build 16299)<br>
Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz, 3408 MHz, 4 Cores</p>
</li>
<li>
<p dir="auto"><strong>TensorFlow installed from (source or binary)</strong>:<br>
binary</p>
</li>
<li>
<p dir="auto"><strong>TensorFlow version</strong>:<br>
tensorflow-gpu 1.5.0, 1.7.0</p>
</li>
<li>
<p dir="auto"><strong>Python version</strong>:<br>
3.5.5 &amp; 3.6 (via anaconda, conda 4.5.1.)</p>
</li>
<li>
<p dir="auto"><strong>Bazel Version</strong>:<br>
N/A</p>
</li>
<li>
<p dir="auto"><strong>CUDA/cuDNN version</strong>:<br>
Tested combinations:<br>
CUDA 9.0 and CuDNN 7.1.2 (tested on tensorflow 1.5.0, 1.7.0 and 1.8.0-dev20180329)<br>
CUDA 9.1 and CuDNN 7.0.5 (tested on tensorflow 1.5.0 and 1.7.0)</p>
</li>
<li>
<p dir="auto"><strong>GPU model and memory</strong>:<br>
NVIDIA GeForce GTX 1080 (GP104-400) [Hewlett-Packard], 8192 MBytes of GDDR5X SDRAM [Micron]</p>
</li>
<li>
<p dir="auto"><strong>Exact command to reproduce</strong>:<br>
See: <em>Have I written custom code...</em></p>
</li>
</ul>
<p dir="auto">=================================================================<br>
EDIT:</p>
<p dir="auto">Threadstarter here, hello.</p>
<blockquote>
<p dir="auto">Could you try with the latest nightly?<br>
<a href="https://files.pythonhosted.org/packages/67/c0/e68a4f0400340b54c887703baa8eee188042c3d65a0cf535dda71abffbc2/tf_nightly_gpu-1.13.0.dev20190205-cp37-cp37m-win_amd64.whl" rel="nofollow">https://files.pythonhosted.org/packages/67/c0/e68a4f0400340b54c887703baa8eee188042c3d65a0cf535dda71abffbc2/tf_nightly_gpu-1.13.0.dev20190205-cp37-cp37m-win_amd64.whl</a></p>
</blockquote>
<p dir="auto"><strong>This works!</strong> I checked with that wheel, and then with <code class="notranslate">tf-nightly-gpu-2.0-preview</code> on PYPI, which also worked.<br>
I initially wanted to use the anaconda cudatoolkit and cudnn packages, but currently, cudnn is only available up to version 7.3.1 on anaconda-cloud. Tensorflow 2.0 however, is compiled with 7.4.1, so I had to do this the oldschool way, and download the setups from Nvidia.<br>
Soon, though...<a href="https://imgur.com/a/A2jZizt" rel="nofollow">soon</a>.</p>
<p dir="auto">For everyone, here's what I did, as a guide:</p>
<h3 dir="auto">How to install Tensorflow Nightly 2.0 GPU in Anaconda on Windows 10 x64</h3>
<p dir="auto">• I installed these CUDA/CuDnn Versions:<br>
– cuda_10.0.130_win10_network (Nvidia CUDA Download: <a href="https://developer.nvidia.com/cuda-toolkit" rel="nofollow">https://developer.nvidia.com/cuda-toolkit</a>)<br>
– cuDNN v7.4.1 (Nov 8, 2018), for CUDA 10.0 (Nvidia CuDnn Download: <a href="https://developer.nvidia.com/cudnn" rel="nofollow">https://developer.nvidia.com/cudnn</a>)<br>
– Don't forget to check, whether the Cuda setup has correctly written itself to the PATH system variable.<br>
– Reboot.<br>
• Now make a new environment in Anaconda and activate it:<br>
– <code class="notranslate">conda create --name tf2-nightly-gpu python=3.6</code><br>
– <code class="notranslate">activate tf2-nightly-gpu</code><br>
• Now, with the new env still activated, install the latest Tensorflow 2.0 nightly GPU build from PYPI:<br>
– <code class="notranslate">pip install tf-nightly-gpu-2.0-preview</code><br>
• For machine learning in Jupyter notebook (or Jupyter Lab) , you need these as well:<br>
– <code class="notranslate">conda install nb_conda matplotlib scipy Pillow pandas scikit-learn</code><br>
• Check, if your GPU is recognized by Tensorflow. Open the Anaconda prompt, activate the new environment and type <code class="notranslate">python</code>, then press Enter. Now type:<br>
<code class="notranslate">import tensorflow as tf</code><br>
<code class="notranslate">tf.test.is_gpu_available(cuda_only=False,min_cuda_compute_capability=None) </code><br>
• Output should be something like this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="(tf2-nightly-gpu) C:\Users\___&gt;python
&gt;&gt;&gt; import tensorflow as tf
&gt;&gt;&gt; tf.test.is_gpu_available(cuda_only=False,min_cuda_compute_capability=None)
2019-03-19 17:46:25.722209: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2019-03-19 17:46:25.729724: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library nvcuda.dll
2019-03-19 17:46:25.922934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1551] Found device 0 with properties:
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7335
pciBusID: 0000:01:00.0
totalMemory: 8.00GiB freeMemory: 6.61GiB
2019-03-19 17:46:25.938231: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1674] Adding visible gpu devices: 0
2019-03-19 17:46:26.539185: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1082] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-19 17:46:26.546009: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1088]      0
2019-03-19 17:46:26.550123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1101] 0:   N
2019-03-19 17:46:26.554188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1222] Created TensorFlow device (/device:GPU:0 with 6360 MB memory) -&gt; physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
True"><pre class="notranslate"><code class="notranslate">(tf2-nightly-gpu) C:\Users\___&gt;python
&gt;&gt;&gt; import tensorflow as tf
&gt;&gt;&gt; tf.test.is_gpu_available(cuda_only=False,min_cuda_compute_capability=None)
2019-03-19 17:46:25.722209: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2019-03-19 17:46:25.729724: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library nvcuda.dll
2019-03-19 17:46:25.922934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1551] Found device 0 with properties:
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7335
pciBusID: 0000:01:00.0
totalMemory: 8.00GiB freeMemory: 6.61GiB
2019-03-19 17:46:25.938231: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1674] Adding visible gpu devices: 0
2019-03-19 17:46:26.539185: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1082] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-19 17:46:26.546009: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1088]      0
2019-03-19 17:46:26.550123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1101] 0:   N
2019-03-19 17:46:26.554188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1222] Created TensorFlow device (/device:GPU:0 with 6360 MB memory) -&gt; physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
True
</code></pre></div>
<p dir="auto">• Done.</p>