tensorflow docker on WSL2: failed call to cuInit: CUDA_ERROR_NO_DEVICE
Hi @jw-redpanda ,

To enable GPU support on docker environment first we need to run nvidia-docker first as per Tf documentation [here](https://www.tensorflow.org/install/docker#gpu_support). Please have a look into the Nvidia documentation [here](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/user-guide.html) for more details.

I am attaching the logs below where I have used docker with GPU support earlier. Please go through the logs and it may be of some help. Thanks!

[docker_with _gpu-logs.txt](https://github.com/tensorflow/tensorflow/files/11350348/docker_with._gpu-logs.txt)


 Hi @jw-redpanda ,
You are using cpu tensorflow image,
try using this

`
docker run -it --rm --gpus all tensorflow/tensorflow:2.12.0-gpu \
    python -c 'import tensorflow as tf; gpus = tf.config.list_physical_devices("GPU"); print(f">>>> GPUs={gpus}")'
`

If this doesn't work check your GPU compatibility and then check which tensorflow version your GPU is compatible with. @jenil111 Thanks for your info.

I tried out your suggestion, using the image `tensorflow/tensorflow:2.12.0-gpu`, instead of `tensorflow/tensorflow:latest-gpu`, but it still reports the same error `CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected`.  Meanwhile, I also run nvidia driver utility `nvidia-smi`.  It is able to find the GPU and display the info when running in both container images.
```
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.47                 Driver Version: 531.68       CUDA Version: 11.8     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2070         On | 00000000:01:00.0  On |                  N/A |
|  0%   46C    P8               11W / 185W|    287MiB /  8192MiB |      7%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
```

If the `nvidia-smi` is able to display the driver and CUDA version info, why tensorflow is not able to find the GPU in the container? 

Anyone using WSL2 tensorflow gpu image can help?  May be I miss some configurations.

 I have used 'tensorflow/tensorflow:2.12.0-gpu'  image on rtx 3060 and it's working fine.

I think rtx 2070 doesn't support cuda 11.8 can you check on cuda 10 version tensorflow image 
 @SuryanarayanaY Thank you. I guess I have already followed the official installation guidelines you pointed out.

I tried out three cases to compare the results:
1. WSL2/Ubuntu 18.04 (Windows host) + docker container `tensorflow/tensorflow:latest-gpu` + NVIDIA Container Runtime installed/configured
`CUDA_ERROR_NO_DEVICE` but `nvidia-smi` can display driver/CUDA info
2. WSL2/Ubuntu 18.04 (without docker container) + system installed cuDNN/CUDA12.1
<b>NOTE: using the same WSL2 as in case 1, i.e. using WSL2 and CUDA directly</b>
`GPU is found`
3. Ubuntu 16.04 (Linux host) +  docker container `tensorflow/tensorflow:latest-gpu` + NVIDIA Container Runtime installed/configured
`GPU is found`




 > 



> I have used 'tensorflow/tensorflow:2.12.0-gpu' image on rtx 3060 and it's working fine.
> 
> I think rtx 2070 doesn't support cuda 11.8 can you check on cuda 10 version tensorflow image

@jenil111 Thanks for the info.  
Some hints are seen.  As per your suggestion, I tried out a tf version that uses cuda 10, e.g. `tensorflow/tensorflow:2.3.4-gpu`

```
root@7fe5fcc3cd6c:/# nvidia-smi
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.47                 Driver Version: 531.68       CUDA Version: 10.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2070         On | 00000000:01:00.0 Off |                  N/A |
|  0%   41C    P8                7W / 185W|    305MiB /  8192MiB |      4%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
```

```
root@7fe5fcc3cd6c:/# python
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2023-04-28 15:22:28.342207: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
tf.co>>> tf.config.list_physical_devices('GPU')
2023-04-28 15:22:36.768408: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2023-04-28 15:22:36.774718: E tensorflow/stream_executor/cuda/cuda_driver.cc:314] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is det
ected
2023-04-28 15:22:36.774822: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (7fe5fcc3cd6
c): /proc/driver/nvidia/version does not exist
```
From the log,  libcudart.so.10.1 and libcuda.so.1 can be loaded.  tf seems to hardcode to read /proc/driver/nvidia/version, but this folder does NOT exist in WSL2 instance.  In a ubuntu linux host, this folder is found.  This seems to be the reason why WSL2 tensorflow docker finds no GPU.

 @jenil111 Hi Jen, since you have the working tensorflow docker environment on WSL2, may I know if your environment has the folder `/proc/driver/nvidia/version` ?  If you have this folder, I need to double-check my installed environment.

Although I am able to use tensorflow + GPU on WSL2 without docker, I still want the docker option since it really simplifies the dependency requirements when using different tensorflow versions that need different cuDNN/CUDA versions. hey @jw-redpanda , I am using 
`
tensorflow/tensorflow:2.12.0-gpu
`
docker image which has cuda 11.8.
and I can see this file "/proc/driver/nvidia/version" look below output
`
root@430e6753b621:~ cat /proc/driver/nvidia/version 
`
`
NVRM version: NVIDIA UNIX x86_64 Kernel Module  530.30.02  Wed Feb 22 04:11:39 UTC 2023
GCC version:  gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04) 
`
 @jenil111 Thank you so much for your help.  I guess you provided important hints for me to troubleshoot the issue. I have finally managed to solve the issue.  It guess my environment was corrupted.  
Sharing the steps below in case someone encounters similar issue.

* Remove all nvidia and cuda packages (Ubuntu/WSL2)
`sudo apt-get purge -y '*cuda*' '*nvidia*'`
* Install Ubuntu WSL2 [driver](https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0&target_type=deb_local)
I tried out the `deb(network)` before but it seemed not working.  Instead, I used the `deb(local)`
* Install nvidia container [toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker)
* Re-install Windows nvidia driver
I selected `perform clean installation`
This seems to be the most important step to rectify the envrionment
* Shutdown and re-run the WSL2
In Windows adminstrative cmd prompt `wsl --shutdown`
* Start docker service (WSL2)
`sudo service docker start`
* Run docker image `tensorflow/tensorflow:2.12.0-gpu`
`docker run -it --gpus all --rm tensorflow/tensorflow:2.12.0-gpu bash`
* Import tensorflow and list GPU
```
root@adcf5339c0c4:/# python
Python 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2023-05-01 14:36:18.705031: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
>>> tf.config.list_physical_devices('GPU')
2023-05-01 14:36:34.143821: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-05-01 14:36:34.148414: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-05-01 14:36:34.148717: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
```
* Check /proc/driver/nvidia folder (NOT found)
```
root@adcf5339c0c4:/# ls -la /proc/driver
total 0
dr-xr-xr-x   4 root root 0 May  1 14:39 .
dr-xr-xr-x 193 root root 0 May  1 14:36 ..
-r--r--r--   1 root root 0 May  1 14:39 nvram
-r--r--r--   1 root root 0 May  1 14:39 rtc
```
 Hi @jw-redpanda ,

Thanks for the confirmation and also for sharing the steps that succeeded installation. Could you please spare some time and mark this as closed if resolved for you. Thanks! Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60429">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60429">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

v2.12.0-rc1-12-g0db597d0d75 2.12.0

### Custom Code

No

### OS Platform and Distribution

Ubuntu 18.04.06 on WSL2

### Mobile device

_No response_

### Python version

Python 3.8.10

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

11.8/8.6

### GPU model and memory

NVIDIA GeForce RTX 2070 / 8GB

### Current Behaviour?

Windows host
* version: 10.0.22621.1555

WSL2/Ubuntu
* Ubuntu 18.04.6 LTS
* Kernel version: 5.10.102.1

NVIDIA Container Toolkit
* CLI version 1.13.1 (commit: 28b70663f1a2b982e59e83bcf1844177dc745208)

Docker (Ubuntu) 
* docker version: 23.0.5, build bc4487a
* image tag: tensorflow/tensorflow:latest-gpu
* image id: cbe3a4f4c2a0
* /etc/docker/daemon.json (already configured the NVIDIA Container Runtime. Docker is restarted to apply change as well)
```json
{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}
```


When running in a container, tensorflow reports no GPU is found.
```
$ docker run -it --rm --gpus all tensorflow/tensorflow:latest-gpu \
    python -c 'import tensorflow as tf; gpus = tf.config.list_physical_devices("GPU"); print(f">>>> GPUs={gpus}")'

2023-04-28 03:19:06.723090: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-28 03:19:08.020353: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:266] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
>>>> GPUs=[]
```

However, `nvidia-smi` is able to print GPU info.
```
$ docker run -it --rm --gpus all tensorflow/tensorflow:latest-gpu nvidia-smi

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.47                 Driver Version: 531.68       CUDA Version: 11.8     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2070         On | 00000000:01:00.0  On |                  N/A |
|  0%   48C    P8               11W / 185W|    175MiB /  8192MiB |      2%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
```

### Standalone code to reproduce the issue

```shell
To reproduce the issue, refer to the `docker run` command described in the Current Behaviour.

Some hints
* When running the same configuration and commands on the Ubuntu host, GPU can be found as expected.
* Under WSL2, without running in a docker container, tensorflow is able to use GPU.
* Strange thing is when running inside the tensorflow container, tensorflow failed to find the GPU. However, nvidia-smi is able to see the GPU.
```


### Relevant log output

_No response_</details>