Tensorflow 2.15 Docker image cannot find the GPU drivers, but nvidia-smi can.
Hello @gmarcusm!
If you are not using TF v2.15 specifically then, could you try to use TF v2.14 which willl work fine with TF GPU drivers. Thank you! > Hello @gmarcusm! If you are not using TF v2.15 specifically then, could you try to use TF v2.14 which willl work fine with TF GPU drivers. Thank you!

Hi ! I use this as part of a CI workflow and I want to test against TF 2.15 specifically, so it does require 2.15 GPU support in this case.

If the `-gpu` docker images are not working as intended, shouldn't they be removed until the issue is fixed? I am using TF 2.15 with cuda 12.3, and I have encountered the same problem that my tf cannot find GPUs while nvidia-smi can. I have the same issue after updated v1.14.3 nvidia-container-toolkit. 
GPU 2.14 and nightly work fine.
```console
williamharris@pop-os:~$ docker run --gpus all -it tensorflow/tensorflow:2.15.0-gpu-jupyter bash

________                               _______________
___  __/__________________________________  ____/__  /________      __
__  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
_  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ /
/_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/


WARNING: You are running this container as root, which can cause new files in
mounted volumes to be created as the root user on your host machine.

To avoid this, run the container by specifying your user's userid:

$ docker run -u $(id -u):$(id -g) args...

root@087d48984615:/tf# nvidia-smi
Tue Nov 21 05:53:18 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.29.02              Driver Version: 545.29.02    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2070 ...    Off | 00000000:01:00.0  On |                  N/A |
|  0%   41C    P8              26W / 235W |    507MiB /  8192MiB |      3%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
root@087d48984615:/tf# python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
2023-11-21 05:53:26.043780: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-11-21 05:53:26.044955: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2023-11-21 05:53:26.062586: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-11-21 05:53:26.062604: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-11-21 05:53:26.063167: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-11-21 05:53:26.066161: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2023-11-21 05:53:26.066261: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-21 05:53:26.636830: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-11-21 05:53:26.637345: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2256] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]
root@087d48984615:/tf# 
``` I think I have the same problem.
GPU: A800 DriveVersion: 525.105.17 CUDA Version: 12.0
I’ve installed such versions below
tensorflow 2.9.0 / 2.12/ 2.13.1 / 2.14.0

But it seems like tensorflow could not detect GPU because:
available = tf.test.is_gpu_available() is_cuda_gpu_available = tf.test.is_gpu_available(cuda_only=True) is_cuda_gpu_min_3 = tf.test.is_gpu_available(True, (3,0)) All were “False”.

When I tried pip install tensorflow[and-cuda]==2.15.0 It exactly returns “True”

But still received the error message:
`E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered` @angerson could you PTAL when able? For using pip to install in Colab, please follow the below step for GPU to be detected in Tensorflow 2.15, till the issue is resolved.

`pip install --extra-index-url https://pypi.nvidia.com tensorflow[and-cuda]` works for upgrading to TF 2.15 and detects GPU in Colab. Hello,

I came across this thread and noticed that I am experiencing a similar issue with TensorFlow 2.15 on my system. I have been unable to get TensorFlow to recognize my GPU, and I thought sharing my setup and steps I've taken might contribute to finding a solution.

Here are the specifics of my setup:

Operating System: Windows 11 Home
Python Version: 3.10.1
TensorFlow Version: 2.15
CUDA Version: 12.3
cuDNN Version: 12
GPU: NVIDIA GeForce RTX 4070 Ti
GPU Driver Version: (Please include your GPU driver version here)
Despite correctly setting up the environment variables and restarting my computer after each configuration, tf.config.list_physical_devices('GPU') returns an empty list, and testing cuDNN installation returns false.

It's interesting to note that this problem persists with TensorFlow 2.15, as I have taken all recommended steps to ensure proper installation and configuration. If anyone has insights or suggestions on what might be going wrong, it would be greatly appreciated.

Thank you for your time and help!

Best regards,
bayerno(helped by GPT 4) I encountered the same problem.
```python
tf.__version__  # '2.15.0'
print(tf.config.list_physical_devices('GPU'))  # []
```

nvidia-smi
```bash
Sat Feb  3 18:53:33 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 546.12                 Driver Version: 546.12       CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4060 ...  WDDM  | 00000000:01:00.0 Off |                  N/A |
| N/A   44C    P0              14W / 115W |      0MiB /  8188MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
```

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

TF 2.15.0

### Custom code

No

### OS platform and distribution

Linux Ubuntu 22.04

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

CUDA 12.2

### GPU model and memory

NVIDIA TITAN V

### Current behavior?

Running TensorFlow 2.15.0 from within Docker image does not find GPU drivers.

nvidia-smi reports the GPUs as available.

This works as intended with a TF 2.14 image in the same machine.


### Standalone code to reproduce the issue

```shell
docker run --gpus all -it tensorflow/tensorflow:2.15.0-gpu-jupyter bash

<within the container>

# nvidia-smi

# python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
```


### Relevant log output

```shell
# nvidia-smi

Thu Nov 16 16:47:22 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA TITAN V                 Off | 00000000:65:00.0 Off |                  N/A |
| 29%   46C    P8              27W / 250W |   1693MiB / 12288MiB |      2%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

# python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

2023-11-16 16:46:54.131081: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2023-11-16 16:46:54.255566: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-11-16 16:46:54.255623: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-11-16 16:46:54.276648: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-11-16 16:46:54.327586: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2023-11-16 16:46:54.328299: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-16 16:46:56.486851: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2256] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]
```
