TensorFlow Not Detecting GPU on Compute Node
# Update on Issue with TensorFlow GPU Detection

After my initial post, I took the following steps to further investigate the issue:

1. ** Replicated the problem with Different cuDNN Version:** Even after pip installing the cuDNN version `8.6.0.163` recommended in TensorFlow's documentation, the problem persisted. TensorFlow still could not detect the GPU.
2. **Module Deactivation Check:** To determine if there was a conflict between the cluster's module system and my conda/pip installations, I deactivated the modules and relied solely on the versions installed via conda and pip.
3. **Custom Environment Variables for Modules:** After confirming the issue persisted without the modules, I then tried setting up the environment using the modules exclusively configuring the paths using those of the module system rather than those provided by the installation instructions:
    - For CUDA: `/home/software/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/cuda-11.8.0-rqftjjg3pwtogsetgcrrytjcqutxgtaj/bin`
    - For cuDNN: `/home/software/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/cudnn-8.7.0.84-11.8-qibz3uecpmz5hiosbohqaniedyu6m6r5/lib`

I adjusted the PATH and LD_LIBRARY_PATH environment variables to prioritize the module system's CUDA and cuDNN installations whenever my specific Conda environment was activated.

Despite these troubleshooting steps, TensorFlow still does not recognize the GPU on the system. I'm continuing to seek insights or suggestions to resolve this issue.

 Check this [Tensorflow doesn't seem to see my gpu](https://stackoverflow.com/questions/41402409/tensorflow-doesnt-seem-to-see-my-gpu) To my understanding tensorflow-gpu has been deprecated since 2022.

Tried downgrading to TensorFlow 2.10 as suggested for windows users even though those problems don't seem super relevant to this case as I am running on Linux. After downgrading my CUDA and cuDNN to 11.2 and 8.1 respectively to no avail. Hi @masonhargrave ,

I think you are missing these 2 steps as mentioned in documentation [instructions](https://www.tensorflow.org/install/pip#step-by-step_instructions) for GPU setup. Please refer the attached source and follow the instructions sequentially.

```
CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))
export LD_LIBRARY_PATH=$CUDNN_PATH/lib:$CONDA_PREFIX/lib/:$LD_LIBRARY_PATH
```

I have followed the same instructions and able to detect GPU.Please refer attached logs.
```
(tf2.13) suryanarayanay@surya-ubuntu20:~$ python3 -c "import tensorflow as tf; print(tf.__version__)"
2023-08-31 06:52:35.206843: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9498] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-08-31 06:52:35.206913: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-08-31 06:52:35.211084: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-08-31 06:52:35.524830: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-31 06:52:37.148708: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
WARNING:tensorflow:From /home/suryanarayanay/miniconda3/envs/tf2.13/lib/python3.11/site-packages/tensorflow/python/ops/distributions/distribution.py:259: ReparameterizationType.__init__ (from tensorflow.python.ops.distributions.distribution) is deprecated and will be removed after 2019-01-01.
Instructions for updating:
The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.
WARNING:tensorflow:From /home/suryanarayanay/miniconda3/envs/tf2.13/lib/python3.11/site-packages/tensorflow/python/ops/distributions/bernoulli.py:165: RegisterKL.__init__ (from tensorflow.python.ops.distributions.kullback_leibler) is deprecated and will be removed after 2019-01-01.
Instructions for updating:
The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.
2.15.0-dev20230813

(tf2.13) suryanarayanay@surya-ubuntu20:~$ nvidia-smi
Thu Aug 31 06:56:15 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   71C    P0    31W /  70W |      2MiB / 15360MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            Off  | 00000000:00:05.0 Off |                    0 |
| N/A   67C    P0    28W /  70W |      2MiB / 15360MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla T4            Off  | 00000000:00:06.0 Off |                    0 |
| N/A   69C    P0    30W /  70W |      2MiB / 15360MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla T4            Off  | 00000000:00:07.0 Off |                    0 |
| N/A   72C    P0    31W /  70W |      2MiB / 15360MiB |      7%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
(tf2.13) suryanarayanay@surya-ubuntu20:~$ 

(tf2.13) suryanarayanay@surya-ubuntu20:~$ python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
2023-08-31 06:54:14.005166: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9498] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-08-31 06:54:14.005226: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-08-31 06:54:14.005283: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-08-31 06:54:14.014421: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-31 06:54:14.824009: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
WARNING:tensorflow:From /home/suryanarayanay/miniconda3/envs/tf2.13/lib/python3.11/site-packages/tensorflow/python/ops/distributions/distribution.py:259: ReparameterizationType.__init__ (from tensorflow.python.ops.distributions.distribution) is deprecated and will be removed after 2019-01-01.
Instructions for updating:
The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.
WARNING:tensorflow:From /home/suryanarayanay/miniconda3/envs/tf2.13/lib/python3.11/site-packages/tensorflow/python/ops/distributions/bernoulli.py:165: RegisterKL.__init__ (from tensorflow.python.ops.distributions.kullback_leibler) is deprecated and will be removed after 2019-01-01.
Instructions for updating:
The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:3', device_type='GPU')]
``` Hi @SuryanarayanaY, thank you for your reply!

As you can see in the *Configuring System Path* section of the *Standalone code to reproduce the issue* I do in fact run those commands from the instructions, except in the more automated way suggested below the commands you are referencing. 

```
# Configuring system paths
mkdir -p $CONDA_PREFIX/etc/conda/activate.d
echo 'CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
echo 'export LD_LIBRARY_PATH=$CUDNN_PATH/lib:$CONDA_PREFIX/lib/:$LD_LIBRARY_PATH' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
```

I reran my code using the non-automated commands you suggested with no change to the problem. This is expected as the commands appear to be redundant and no different than the commands included in my minimal example.

```
(tf-test) [mah4021@scu-node018 mah4021]$ python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
2023-09-02 21:53:49.520842: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-09-02 21:53:49.566393: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-09-02 21:53:50.515064: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-09-02 21:53:51.443662: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:268] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error
2023-09-02 21:53:51.443727: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: scu-node018.scu
2023-09-02 21:53:51.443767: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: scu-node018.scu
2023-09-02 21:53:51.443861: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: 535.104.5
2023-09-02 21:53:51.443921: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: 535.104.5
2023-09-02 21:53:51.443946: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:309] kernel version seems to match DSO: 535.104.5
[]
```

Any other thoughts here? Hi @masonhargrave ,

Could you please confirm the command you used for tensorflow installation?

Please note that from Tf2.14V onwards GPU package available as `tensorflow[and-cuda]`. Please try using the command 
`pip install tensorflow[and-cuda] `. You can refer the documentation [here](https://www.tensorflow.org/install/pip). This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61700">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61700">No</a>

### Issue type

Build/Install

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

binary

### TensorFlow version

v1.12.1-99044-gc6ecfeac886 2.15.0-dev20230825

### Custom code

No

### OS platform and distribution

CentOS Linux release 7.9.2009 (Core)

### Mobile device

_No response_

### Python version

3.9.13

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

11.8.0 / 8.7.0.84

### GPU model and memory

_No response_

### Current behavior?

## Behavior summary 🖥️

I'm trying to set up TensorFlow on a compute node within our scientific computing cluster and am running into a problem. After setting up a new environment and installing TensorFlow, running TensorFlow's `tf.config.list_physical_devices('GPU')` method returns an empty list, indicating no GPU devices are detected. However, `nvidia-smi` shows a Quadro RTX 6000 GPU present on the system. The TensorFlow CPU validation works without issues, but the GPU validation does not.

## Possible Areas of Concern 🚩

1. **Modules vs. Conda/Pip Installations:** I've loaded specific versions of CUDA and cuDNN using the cluster's module system. However, I also used conda and pip to install these within my environment. Could this mixed approach cause any conflicts for TensorFlow? I'm unsure which version TensorFlow might prioritize or if it would create any confusion.
2. **CUDA Version Mismatch?:** When I run `nvidia-smi`, it indicates a CUDA version of 12.2. Yet, I've loaded and installed a CUDA version of 11.8 using both modules and conda. I'm wondering if this difference could lead to any issues. Does TensorFlow need a strict match with the CUDA version?
3. **Different cuDNN Version:** The TensorFlow installation guide mentioned cuDNN version 8.6.0.163, but I installed 8.7.0.84 for what I thought might be better consistency with the modules I loaded. Could this version difference be problematic? Is TensorFlow particular about the cuDNN version it works with?

I'm looking for insights from anyone who might have navigated similar issues or can provide clarity on these points. Also of course if anyone knows how I might go about further troubleshooting/diagnosing this that'd be great. Thanks for the help!  

### Standalone code to reproduce the issue

```shell
# Setting up a new conda environment with python 3.9
conda create --name myenv python=3.9
conda activate myenv

# Running NVIDIA's System Management Interface to check GPU
nvidia-smi

# Loading necessary modules for CUDA/cuDNN
module load cuda-11.8.0-gcc-8.2.0-rqftjjg
module load cudnn-8.7.0.84-11.8-gcc-8.2.0-qibz3ue

# Installing CUDA Toolkit and cuDNN using Conda and Pip
conda install -c conda-forge cudatoolkit=11.8.0
pip install nvidia-cudnn-cu11==8.7.0.84

# Configuring system paths
mkdir -p $CONDA_PREFIX/etc/conda/activate.d
echo 'CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
echo 'export LD_LIBRARY_PATH=$CUDNN_PATH/lib:$CONDA_PREFIX/lib/:$LD_LIBRARY_PATH' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh

# Upgrading pip and installing TensorFlow
pip install --upgrade pip
pip install tensorflow==2.13.*

# Running CPU validation
python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"

# Running GPU validation
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
```


### Relevant log output

```shell
# CPU validation
2023-08-25 14:11:07.056941: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-08-25 14:11:07.105836: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-25 14:11:08.620525: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-08-25 14:11:10.459520: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:268] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error
2023-08-25 14:11:10.459591: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: scu-node018.scu
2023-08-25 14:11:10.459608: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: scu-node018.scu
2023-08-25 14:11:10.459712: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: 535.104.5
2023-08-25 14:11:10.459783: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: 535.104.5
2023-08-25 14:11:10.459811: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:309] kernel version seems to match DSO: 535.104.5
tf.Tensor(-964.9978, shape=(), dtype=float32)

# GPU validation
2023-08-25 14:11:35.537281: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-08-25 14:11:35.582391: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-25 14:11:36.286683: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-08-25 14:11:36.956393: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:268] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error
2023-08-25 14:11:36.956458: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: scu-node018.scu
2023-08-25 14:11:36.956495: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: scu-node018.scu
2023-08-25 14:11:36.956591: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: 535.104.5
2023-08-25 14:11:36.956655: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: 535.104.5
2023-08-25 14:11:36.956683: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:309] kernel version seems to match DSO: 535.104.5
[]

# nvidia-smi Outpu
Fri Aug 25 13:59:33 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro RTX 6000                Off | 00000000:AF:00.0 Off |                    0 |
| N/A   31C    P0              54W / 250W |      0MiB / 23040MiB |      5%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
```
