title: How to speed up the 'Adding visible gpu devices' process in tensorflow with a 30 series card?
body: <p>I get stuck with that for ~2 minute every time I run the code. Many people on the Internet said that it would only take a long time in the first run, but that's not my case. Although it doesn't make anything go wrong, it's pretty annoying. When I'm stuck, the system is under pretty low usage, including the CPU, system RAM, GPU, video memory.
  I'm using Nvidia Geforce RTX 3070, Windows 10 x64 20H2.Here's my environment:</p>
  <pre><code># Name                    Version                   Build  Channel
  blas                      1.0                         mkl    defaults
  boto3                     1.16.47                  pypi_0    pypi
  botocore                  1.19.47                  pypi_0    pypi
  ca-certificates           2020.12.8            haa95532_0    defaults
  certifi                   2020.12.5        py38haa95532_0    defaults
  click                     7.1.2                    pypi_0    pypi
  cudatoolkit               11.0.221             h74a9793_0    defaults
  freetype                  2.10.4               hd328e21_0    defaults
  intel-openmp              2020.2                      254    defaults
  jmespath                  0.10.0                   pypi_0    pypi
  joblib                    1.0.0                    pypi_0    pypi
  jpeg                      9b                   hb83a4c4_2    defaults
  keras                     2.4.3                    pypi_0    pypi
  libpng                    1.6.37               h2a8f88b_0    defaults
  libtiff                   4.1.0                h56a325e_1    defaults
  libuv                     1.40.0               he774522_0    defaults
  lz4-c                     1.9.2                hf4a77e7_3    defaults
  mkl                       2020.2                      256    defaults
  mkl-service               2.3.0            py38h196d8e1_0    defaults
  mkl_fft                   1.2.0            py38h45dec08_0    defaults
  mkl_random                1.1.1            py38h47e9c7a_0    defaults
  ninja                     1.10.2           py38h6d14046_0    defaults
  numpy                     1.19.2           py38hadc3359_0    defaults
  numpy-base                1.19.2           py38ha3acd2a_0    defaults
  olefile                   0.46                       py_0    defaults
  openssl                   1.1.1i               h2bbff1b_0    defaults
  pillow                    8.0.1            py38h4fa10fc_0    defaults
  pip                       20.3.3           py38haa95532_0    defaults
  python                    3.8.5                h5fd99cc_1    defaults
  pytorch                   1.7.1           py3.8_cuda110_cudnn8_0    pytorch
  regex                     2020.11.13               pypi_0    pypi
  s3transfer                0.3.3                    pypi_0    pypi
  sacremoses                0.0.43                   pypi_0    pypi
  scikit-learn              0.24.0                   pypi_0    pypi
  scipy                     1.6.0                    pypi_0    pypi
  sentencepiece             0.1.94                   pypi_0    pypi
  setuptools                51.0.0           py38haa95532_2    defaults
  six                       1.15.0           py38haa95532_0    defaults
  sklearn                   0.0                      pypi_0    pypi
  sqlite                    3.33.0               h2a8f88b_0    defaults
  tb-nightly                2.5.0a20210101           pypi_0    pypi
  threadpoolctl             2.1.0                    pypi_0    pypi
  thulac                    0.2.1                    pypi_0    pypi
  tk                        8.6.10               he774522_0    defaults
  torchaudio                0.7.2                      py38    pytorch
  torchvision               0.8.2                py38_cu110    pytorch
  transformers              2.1.1                    pypi_0    pypi
  typing_extensions         3.7.4.3                    py_0    defaults
  vc                        14.2                 h21ff451_1    defaults
  vs2015_runtime            14.27.29016          h5e58377_2    defaults
  wheel                     0.36.2             pyhd3eb1b0_0    defaults
  wincertstore              0.2                      py38_0    defaults
  xz                        5.2.5                h62dcd97_0    defaults
  zlib                      1.2.11               h62dcd97_4    defaults
  zstd                      1.4.5                h04227a9_0    defaults
  </code></pre>
  <p>Although I'm using PyTorch, it's tensorflow rather than PyTorch to blame(according to the logs). I got the same issue with pure tensorflow 2.3</p>
  <pre><code>2021-01-03 01:17:50.516100: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
  2021-01-03 01:17:52.622054: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
  2021-01-03 01:17:52.645796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
  pciBusID: 0000:0a:00.0 name: GeForce RTX 3070 computeCapability: 8.6
  coreClock: 1.725GHz coreCount: 46 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
  2021-01-03 01:17:52.645998: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
  2021-01-03 01:17:52.649575: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
  2021-01-03 01:17:52.649707: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
  2021-01-03 01:17:52.649827: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
  2021-01-03 01:17:52.649928: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
  2021-01-03 01:17:52.651954: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
  2021-01-03 01:17:52.660165: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
  2021-01-03 01:17:52.660416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
  2021-01-03 01:17:52.660971: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2
  To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
  2021-01-03 01:17:52.668967: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x19659fe67d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
  2021-01-03 01:17:52.669132: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
  2021-01-03 01:17:52.669395: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
  pciBusID: 0000:0a:00.0 name: GeForce RTX 3070 computeCapability: 8.6
  coreClock: 1.725GHz coreCount: 46 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
  2021-01-03 01:17:52.669576: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
  2021-01-03 01:17:52.669683: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
  2021-01-03 01:17:52.669790: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
  2021-01-03 01:17:52.669896: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
  2021-01-03 01:17:52.670072: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
  2021-01-03 01:17:52.670201: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
  2021-01-03 01:17:52.670365: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
  2021-01-03 01:17:52.670542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
  2021-01-03 01:18:37.097681: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
  2021-01-03 01:18:37.097876: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0
  2021-01-03 01:18:37.098025: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
  2021-01-03 01:18:37.098301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6591 MB memory) -&gt; physical GPU (device: 0, name: GeForce RTX 3070, pci bus id: 0000:0a:00.0, compute capability: 8.6)
  2021-01-03 01:18:37.101296: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1960330d0d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
  2021-01-03 01:18:37.101474: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce RTX 3070, Compute Capability 8.6
  args:
  Namespace(articles_per_title=10, device='0,1,2,3', length=1000, model_config='config/model_config_small.json', model_path='model/final_model', no_wordpiece=False, repetition_penalty=1.0, save_path='generated/', segment=False, temperature=2.0, titles='我', titles_file='', tokenizer_path='cache/vocab_small.txt', topk=10, topp=0)
  </code></pre>
  <p>I noticed that <a href="https://www.tensorflow.org/install/gpu?" rel="nofollow noreferrer">the tensorflow installation guide for GPU users</a> said that GPUs with Ampere architecture may encounter this issue and can solve that by using <code>export CUDA_CACHE_MAXSIZE=2147483648</code> to expand the default JIT cache. It doesn't work with Windows. I searched my environment variables and none of them named<code>CUDA_CACHE_MAXSIZE</code>.
  I tried adding that on my own, but it still takes a long time to pass <code>Adding Visible Devices 0</code>.
  What should I do?</p>