title: TensorFlow ignores the RTX 3000 series GPU
body: <p>I am trying to train my model using the RTX 3090 GPU.<br />
  In order to be able to use it at all, i had to install <code>TensorFlow==2.4.0-rc0</code>, however, there is a problem with actually using that GPU.</p>
  <p>(Yes, i have downclocked memory as it is getting really toasty while running at stock 19,5 Ghz, that is why memory bandwidth is 60 Gbps lower)</p>
  <p>First of all, it detects GPU but then saying:</p>
  <pre><code>tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
  pciBusID: 0000:01:00.0 name: GeForce RTX 3090 computeCapability: 8.6
  coreClock: 1.695GHz coreCount: 82 deviceMemorySize: 24.00GiB deviceMemoryBandwidth: 871.81GiB/s
  </code></pre>
  <p>Then it says:</p>
  <pre><code>Adding visible gpu devices: 0
  </code></pre>
  <p>But a couple of lines below that message, this message is displayed:</p>
  <pre><code>Created TensorFlow device 
  (/job:localhost/replica:0/task:0/device:GPU:0 with 21821 MB memory) -&gt; 
  physical GPU (device: 0, name: GeForce RTX 3090, pci bus id: 0000:01:00.0, compute capability: 8.6)
  </code></pre>
  <p>And then it just continues to hammer CPU and not actually using GPU at all. The most important part, when training is done purely on CPU, time to complete one epoch is around 80 seconds, however, when GPU is used, it wont be able to complete even a single epoch.</p>
  <p><a href="https://i.stack.imgur.com/GFMYV.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/GFMYV.png" alt="enter image description here" /></a></p>
  <p><strong>This is the complete text output of my Jupyter Notebook (when it is running)</strong></p>
  <pre class="lang-sh prettyprint-override"><code>[I 04:06:47.194 NotebookApp] Kernel started: e4bec12d-3d85-4019-9b5a-67d34a45acfc
  [I 04:06:50.799 NotebookApp] Starting buffering for e4bec12d-3d85-4019-9b5a-67d34a45acfc:591585a545fe4d33977dac034060b33c
  [I 04:06:51.031 NotebookApp] Kernel restarted: e4bec12d-3d85-4019-9b5a-67d34a45acfc
  [I 04:06:51.557 NotebookApp] Restoring connection for e4bec12d-3d85-4019-9b5a-67d34a45acfc:591585a545fe4d33977dac034060b33c
  [I 04:06:51.558 NotebookApp] Replaying 3 buffered messages
  2020-11-06 04:06:53.766169: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
  2020-11-06 04:07:01.412837: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
  2020-11-06 04:07:01.420283: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
  2020-11-06 04:07:01.438547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
  pciBusID: 0000:01:00.0 name: GeForce RTX 3090 computeCapability: 8.6
  coreClock: 1.695GHz coreCount: 82 deviceMemorySize: 24.00GiB deviceMemoryBandwidth: 871.81GiB/s
  2020-11-06 04:07:01.438675: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
  2020-11-06 04:07:01.450544: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
  2020-11-06 04:07:01.450698: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
  2020-11-06 04:07:01.453610: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
  2020-11-06 04:07:01.454496: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
  2020-11-06 04:07:01.457436: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
  2020-11-06 04:07:01.459702: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
  2020-11-06 04:07:01.460296: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
  2020-11-06 04:07:01.460439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
  2020-11-06 04:07:01.461093: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2
  To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
  2020-11-06 04:07:01.461751: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
  pciBusID: 0000:01:00.0 name: GeForce RTX 3090 computeCapability: 8.6
  coreClock: 1.695GHz coreCount: 82 deviceMemorySize: 24.00GiB deviceMemoryBandwidth: 871.81GiB/s
  2020-11-06 04:07:01.461854: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
  2020-11-06 04:07:01.462144: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
  2020-11-06 04:07:01.462407: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
  2020-11-06 04:07:01.462690: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
  2020-11-06 04:07:01.462941: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
  2020-11-06 04:07:01.464597: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
  2020-11-06 04:07:01.464843: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
  2020-11-06 04:07:01.465087: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
  2020-11-06 04:07:01.465348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
  2020-11-06 04:07:01.838515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
  2020-11-06 04:07:01.838596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
  2020-11-06 04:07:01.838999: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
  2020-11-06 04:07:01.839431: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21821 MB memory) -&gt; physical GPU (device: 0, name: GeForce RTX 3090, pci bus id: 0000:01:00.0, compute capability: 8.6)
  2020-11-06 04:07:01.842196: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
  2020-11-06 04:07:10.441807: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
  2020-11-06 04:07:11.435159: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
  2020-11-06 04:07:12.026347: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
  2020-11-06 04:07:12.044635: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
  [I 04:08:47.169 NotebookApp] Saving file at /train_model.ipynb
  2020-11-06 04:13:24.212460: I tensorflow/stream_executor/cuda/cuda_blas.cc:1838] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
  </code></pre>
  <p><strong>P.S. Update #1</strong><br />
  It took 579 seconds to complete single epoch using GPU, while it used to take only 80 seconds to complete it on CPU</p>