CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
Closed as repository I was using was manually setting CUDA_VISIBLE_DEVICES to -1... Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62494">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62494">No</a>

### Issue type

Build/Install

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

unknown 2.4.0

### Custom code

Yes

### OS platform and distribution

Linux, Rocky linux

### Mobile device

_No response_

### Python version

Python 3.8.6

### Bazel version

_No response_

### GCC/compiler version

gcc (GCC) 10.2.0

### CUDA/cuDNN version

CUDA 11.1 cuDNN/8.0.4.30

### GPU model and memory

Nvidia A100

### Current behavior?

Hello!

I am working on a deep learning project.

I have configured Tensorflow correctly and it detects my GPUs when running 

 python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

Results in

`
2023-11-28 16:53:38.531363: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2023-11-28 16:53:44.264370: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2023-11-28 16:53:44.269638: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:53:44.297140: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:03:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
2023-11-28 16:53:44.297732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties: 
pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
2023-11-28 16:53:44.297762: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2023-11-28 16:53:44.363331: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2023-11-28 16:53:44.363424: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2023-11-28 16:53:44.376801: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2023-11-28 16:53:44.380587: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2023-11-28 16:53:44.446207: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2023-11-28 16:53:44.458537: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2023-11-28 16:53:44.460103: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2023-11-28 16:53:44.462356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU')]
`

However, when I run my program I get the following message: 
`
2023-11-28 16:42:02.209699: I tensorflow/stream_executor/platform/default/[dso_loader.cc:49](http://dso_loader.cc:49/)] Successfully opened dynamic library [libcuda.so](http://libcuda.so/).1
2023-11-28 16:42:02.417037: E tensorflow/stream_executor/cuda/[cuda_driver.cc:328](http://cuda_driver.cc:328/)] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
`

My CUDA_VISIBLE_DEVICES is set to "0,1" so it should see both the GPUs but tensorflow doesn’t detect them.

Have anyone else encountered this issue?


### Standalone code to reproduce the issue

```shell
Pip packages

Package                       Version       Editable project location
----------------------------- ------------- -------------------------------------------------
absl-py                       0.10.0
alabaster                     0.7.12
appdirs                       1.4.4
array-record                  0.4.0
asn1crypto                    1.4.0
astunparse                    1.6.3
atomicwrites                  1.4.0
attrs                         20.2.0
Babel                         2.8.0
bcrypt                        3.2.0
bitstring                     3.1.7
blist                         1.3.6
Bottleneck                    1.3.2
CacheControl                  0.12.6
cachetools                    4.2.0
cachy                         0.3.0
ccimport                      0.3.7
certifi                       2020.6.20
cffi                          1.14.3
chardet                       3.0.4
cleo                          0.8.1
click                         7.1.2
clikit                        0.6.2
cloudpickle                   3.0.0
colorama                      0.4.3
contourpy                     1.1.1
crashtest                     0.3.1
cryptography                  3.1.1
cumm-cu111                    0.2.9
cycler                        0.12.1
Cython                        0.29.21
dacite                        1.8.1
dask                          2023.3.1
dataclass_array               1.4.1
deap                          1.3.1
decorator                     4.4.2
dill                          0.3.3
distlib                       0.3.1
dm-tree                       0.1.8
docopt                        0.6.2
docutils                      0.16
easydict                      1.11
ecdsa                         0.16.0
einops                        0.7.0
einsum                        0.3.0
etils                         1.3.0
filelock                      3.0.12
fire                          0.5.0
flatbuffers                   1.12
flit                          3.0.0
flit_core                     3.0.0
fonttools                     4.45.1
fsspec                        0.8.4
future                        0.18.2
gast                          0.3.3
google-auth                   1.24.0
google-auth-oauthlib          0.4.2
google-pasta                  0.2.0
googleapis-common-protos      1.61.0
grpcio                        1.32.0
gviz-api                      1.9.0
h5py                          2.10.0
html5lib                      1.1
idna                          2.10
imageio                       2.33.0
imagesize                     1.2.0
immutabledict                 2.2.0
importlib-metadata            6.8.0
importlib-resources           6.1.1
iniconfig                     1.0.1
install                       1.3.5
intervaltree                  3.1.0
intreehooks                   1.0
ipaddress                     1.0.23
jeepney                       0.4.3
Jinja2                        2.11.2
joblib                        1.3.2
jsonschema                    3.2.0
keras                         2.11.0
Keras-Preprocessing           1.1.2
keyring                       21.4.0
keyrings.alt                  4.0.0
kiwisolver                    1.4.5
lark                          1.1.8
lazy_loader                   0.3
liac-arff                     2.5.0
libclang                      16.0.6
llvmlite                      0.39.1
locket                        1.0.0
lockfile                      0.12.2
Markdown                      3.3.3
MarkupSafe                    2.1.3
matplotlib                    3.6.1
mock                          4.0.2
more-itertools                8.5.0
mpi4py                        3.0.3
mpmath                        1.1.0
msgpack                       1.0.0
netaddr                       0.8.0
netifaces                     0.10.9
networkx                      3.1
ninja                         1.11.1.1
nose                          1.3.7
numba                         0.56.4
numexpr                       2.7.1
numpy                         1.19.4
oauthlib                      3.1.0
opencv-python                 4.8.1.78
OpenEXR                       1.3.9
opt-einsum                    3.3.0
packaging                     23.2
pandas                        1.1.4
paramiko                      2.7.2
partd                         1.4.1
pastel                        0.2.1
pathlib2                      2.3.5
paycheck                      1.0.2
pbr                           5.5.0
pccm                          0.3.4
pcdet                         0.5.2+82e7c4f /cluster/home/martiiv/DeepLearningProject/3DTrans
pexpect                       4.8.0
Pillow                        9.2.0
pip                           20.2.3
pkginfo                       1.5.0.1
plotly                        5.13.1
pluggy                        0.13.1
poetry                        1.1.3
poetry-core                   1.0.0
portalocker                   2.8.2
portpicker                    1.3.1
promise                       2.3
protobuf                      3.14.0
psutil                        5.7.2
ptyprocess                    0.6.0
py                            1.9.0
py-expression-eval            0.3.10
pyarrow                       10.0.0
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pybind11                      2.6.0
pycparser                     2.20
pycrypto                      2.6.1
Pygments                      2.7.1
pylev                         1.3.0
PyNaCl                        1.4.0
pyparsing                     2.4.7
pyrsistent                    0.17.3
pytest                        6.1.1
python-dateutil               2.8.1
pytoml                        0.1.21
pytz                          2020.1
PyWavelets                    1.4.1
PyYAML                        5.3.1
regex                         2020.10.11
requests                      2.24.0
requests-oauthlib             1.3.0
requests-toolbelt             0.9.1
rsa                           4.7
scandir                       1.10.0
scikit-image                  0.19.3
scikit-learn                  1.2.2
scipy                         1.5.4
SecretStorage                 3.1.2
setuptools                    67.6.0
setuptools-scm                4.1.2
SharedArray                   3.1.0
shellingham                   1.3.2
simplegeneric                 0.8.1
simplejson                    3.17.2
six                           1.15.0
snowballstemmer               2.0.0
sortedcontainers              2.2.2
spconv-cu111                  2.1.25
Sphinx                        3.2.1
sphinx-bootstrap-theme        0.7.1
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        1.0.3
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.4
sphinxcontrib-websupport      1.2.4
tabulate                      0.8.7
tblib                         1.7.0
tenacity                      8.2.3
tensorboard                   2.4.0
tensorboard-data-server       0.6.1
tensorboard-plugin-profile    2.4.0
tensorboard-plugin-wit        1.8.0
tensorboardX                  2.6
tensorflow                    2.4.0
tensorflow-estimator          2.4.0
termcolor                     1.1.0
threadpoolctl                 2.1.0
tifffile                      2023.7.10
toml                          0.10.1
tomlkit                       0.7.0
toolz                         0.12.0
torch                         1.8.1+cu111
torchaudio                    0.8.1
torchvision                   0.9.1+cu111
tqdm                          4.66.1
trimesh                       4.0.5
typeguard                     2.13.3
typing-extensions             3.7.4.3
ujson                         4.0.1
urllib3                       1.25.10
virtualenv                    20.0.34
visu3d                        1.5.1
waymo-open-dataset-tf-2-4-0   1.4.1
wcwidth                       0.2.5
webencodings                  0.5.1
Werkzeug                      1.0.1
wheel                         0.35.1
wrapt                         1.12.1
xlrd                          1.2.0
zipp                          3.3.0
```


### Relevant log output

```shell
---------------The waymo sample interval is 1, total sequecnes is 798-----------------

  0%|          | 0/798 [00:00<?, ?it/s]2023-11-28 16:42:02.086115: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:42:02.088466: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:42:02.092942: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:42:02.097380: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:42:02.099599: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:42:02.101598: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:42:02.103590: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:42:02.105583: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:42:02.107602: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:42:02.109606: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:42:02.160116: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:42:02.200029: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:42:02.209699: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:42:02.417037: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-11-28 16:42:02.423311: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:42:02.426811: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: idun-06-18
2023-11-28 16:42:02.427445: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: idun-06-18
2023-11-28 16:42:02.427608: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 515.43.4
2023-11-28 16:42:02.427668: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 515.43.4
2023-11-28 16:42:02.427696: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 515.43.4
2023-11-28 16:42:02.553576: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-11-28 16:42:02.553649: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: idun-06-18
2023-11-28 16:42:02.553683: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: idun-06-18
2023-11-28 16:42:02.553902: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 515.43.4
2023-11-28 16:42:02.553961: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 515.43.4
2023-11-28 16:42:02.553987: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 515.43.4
2023-11-28 16:42:02.563455: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-11-28 16:42:02.563537: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: idun-06-18
2023-11-28 16:42:02.563584: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: idun-06-18
2023-11-28 16:42:02.563781: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 515.43.4
2023-11-28 16:42:02.563848: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 515.43.4
2023-11-28 16:42:02.563877: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 515.43.4
2023-11-28 16:42:02.573007: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-11-28 16:42:02.573069: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: idun-06-18
2023-11-28 16:42:02.573098: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: idun-06-18
2023-11-28 16:42:02.573242: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 515.43.4
2023-11-28 16:42:02.573306: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 515.43.4
2023-11-28 16:42:02.573333: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 515.43.4
2023-11-28 16:42:02.582579: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-11-28 16:42:02.582642: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: idun-06-18
2023-11-28 16:42:02.582670: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: idun-06-18
2023-11-28 16:42:02.582809: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 515.43.4
2023-11-28 16:42:02.582869: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 515.43.4
2023-11-28 16:42:02.582896: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 515.43.4
2023-11-28 16:42:02.591947: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-11-28 16:42:02.592010: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: idun-06-18
2023-11-28 16:42:02.592040: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: idun-06-18
2023-11-28 16:42:02.592180: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 515.43.4
2023-11-28 16:42:02.592258: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 515.43.4
2023-11-28 16:42:02.592287: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 515.43.4
2023-11-28 16:42:02.601424: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-11-28 16:42:02.601488: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: idun-06-18
2023-11-28 16:42:02.601517: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: idun-06-18
2023-11-28 16:42:02.601656: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 515.43.4
2023-11-28 16:42:02.601715: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 515.43.4
2023-11-28 16:42:02.601742: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 515.43.4
2023-11-28 16:42:02.610852: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-11-28 16:42:02.610938: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: idun-06-18
2023-11-28 16:42:02.610969: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: idun-06-18
2023-11-28 16:42:02.611111: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 515.43.4
2023-11-28 16:42:02.611171: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 515.43.4
2023-11-28 16:42:02.611198: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 515.43.4
2023-11-28 16:42:02.632051: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-11-28 16:42:02.632119: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: idun-06-18
2023-11-28 16:42:02.632149: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: idun-06-18
2023-11-28 16:42:02.632301: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 515.43.4
2023-11-28 16:42:02.632363: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 515.43.4
2023-11-28 16:42:02.632390: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 515.43.4
2023-11-28 16:42:02.641638: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-11-28 16:42:02.641728: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: idun-06-18
2023-11-28 16:42:02.641775: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: idun-06-18
2023-11-28 16:42:02.642013: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 515.43.4
2023-11-28 16:42:02.642080: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 515.43.4
2023-11-28 16:42:02.642110: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 515.43.4
2023-11-28 16:42:02.651328: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-11-28 16:42:02.651395: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: idun-06-18
2023-11-28 16:42:02.651425: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: idun-06-18
2023-11-28 16:42:02.651568: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 515.43.4
2023-11-28 16:42:02.651628: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 515.43.4
2023-11-28 16:42:02.651655: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 515.43.4
2023-11-28 16:42:02.669139: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:42:02.713192: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-11-28 16:42:02.738968: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-11-28 16:42:02.739035: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: idun-06-18
2023-11-28 16:42:02.739065: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: idun-06-18
2023-11-28 16:42:02.739212: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 515.43.4
2023-11-28 16:42:02.739283: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 515.43.4
2023-11-28 16:42:02.739310: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 515.43.4
2023-11-28 16:42:02.774039: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-11-28 16:42:02.774103: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: idun-06-18
2023-11-28 16:42:02.774132: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: idun-06-18
2023-11-28 16:42:02.774284: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 515.43.4
2023-11-28 16:42:02.774344: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 515.43.4
2023-11-28 16:42:02.774370: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 515.43.4
2023-11-28 16:42:02.790868: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-11-28 16:42:02.790970: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: idun-06-18
2023-11-28 16:42:02.791020: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: idun-06-18
2023-11-28 16:42:02.791222: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 515.43.4
2023-11-28 16:42:02.791302: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 515.43.4
2023-11-28 16:42:02.791363: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 515.43.4
2023-11-28 16:42:02.820776: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2100205000 Hz
2023-11-28 16:42:02.821129: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2100205000 Hz
2023-11-28 16:42:02.821421: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2100205000 Hz
2023-11-28 16:42:02.821629: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2100205000 Hz
2023-11-28 16:42:02.821832: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2100205000 Hz
2023-11-28 16:42:02.822030: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2100205000 Hz
2023-11-28 16:42:02.822231: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2100205000 Hz
2023-11-28 16:42:02.822442: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2100205000 Hz
2023-11-28 16:42:02.822646: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2100205000 Hz
2023-11-28 16:42:02.822847: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2100205000 Hz
```
