GPU not detected on WSL2
I've gotten the same problem but with python3.12. I found it was a problem with environment variables and finally fixed it. My solution is (if in pwsh):

- Make sure cuda and cudnn are installed correctly.
- Make sure `$Env:CUDA_PATH` is given correctly
- Make sure  `$Env:LD_LIBRARY_PATH` is given correctly
- Make sure `$Env:TF_CUDA_PATHS` is given correctly
- Use `pip install tensorflow[and-cuda]` instead of `pip install tensorflow`. Because the extensions, such as  `nvidia-cuda-*` and  `nvidia-cudnn-*` from pip(Pypi) are also needed. @Zhaopudark  thank you for sharing. what path shoud I set for those environment variables ?
Make sure $Env:CUDA_PATH is given correctly
Make sure $Env:LD_LIBRARY_PATH is given correctly
Make sure $Env:TF_CUDA_PATHS is given correctly > @Zhaopudark thank you for sharing. what path shoud I set for those environment variables ? Make sure $Env:CUDA_PATH is given correctly Make sure $Env:LD_LIBRARY_PATH is given correctly Make sure $Env:TF_CUDA_PATHS is given correctly

That depends on your specific installation method of cuda and cudnn. For me, since I use miniconda and [conda Installation](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#conda-installation), so my setting is as:
```pwsh
# pwsh
# if `conda activate my_env `
# an env variable `$Env:CONDA_PREFIX` will be `/home/your_name/miniconda3/envs/my_env `
# So, the setting is as:
$Env:CUDA_PATH = $Env:CONDA_PREFIX
$Env:LD_LIBRARY_PATH="$Env:CONDA_PREFIX/lib"
$Env:TF_CUDA_PATHS="$Env:CONDA_PREFIX"
```
Actually, either [package manager installation](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#wsl) or [conda Installation](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#conda-installation), the `$Env:CUDA_PATH` should be such a root path where you can see cuda's installed directories, such as `./nvvm`, `./nvvm`, `./extras/Debugger/lib64/libcudacore.a` and so on. And, `$Env:TF_CUDA_PATHS` should be equal to `$Env:CUDA_PATH`.
 > > @Zhaopudark thank you for sharing. what path shoud I set for those environment variables ? Make sure $Env:CUDA_PATH is given correctly Make sure $Env:LD_LIBRARY_PATH is given correctly Make sure $Env:TF_CUDA_PATHS is given correctly
> 
> That depends on your specific installation method of cuda and cudnn. For me, since I use miniconda and [conda Installation](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#conda-installation), so my setting is as:
> 
> ```powershell
> # pwsh
> # if `conda activate my_env `
> # an env variable `$Env:CONDA_PREFIX` will be `/home/your_name/miniconda3/envs/my_env `
> # So, the setting is as:
> $Env:CUDA_PATH = $Env:CONDA_PREFIX
> $Env:LD_LIBRARY_PATH="$Env:CONDA_PREFIX/lib"
> $Env:TF_CUDA_PATHS="$Env:CONDA_PREFIX"
> ```
> 
> Actually, either [package manager installation](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#wsl) or [conda Installation](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#conda-installation), the `$Env:CUDA_PATH` should be such a root path where you can see cuda's installed directories, such as `./nvvm`, `./nvvm`, `./extras/Debugger/lib64/libcudacore.a` and so on. And, `$Env:TF_CUDA_PATHS` should be equal to `$Env:CUDA_PATH`.

Does your setup detect GPU with tensorflow 2.16.1? mine works with 2.15 but not this one. @sh-shahrokhi Yes, since I use python3.12 and only tensorflow-2.16.1 that can be installed by pip. Maybe you should check your cuda and cudnn version that satisfied tensorflow-2.16.1's requirements. 
 @Zhaopudark Thank you for your comment.

Regarding the cudnn and cuda version, I searched around and finally found a matrix [here](https://www.tensorflow.org/install/source#gpu). 

I will update my result once I have installed newer cuda and cudnn. > @Zhaopudark Thank you for your comment.
> 
> Regarding the cudnn and cuda version, I searched around and finally found a matrix [here](https://www.tensorflow.org/install/source#gpu).
> 
> I will update my result once I have installed newer cuda and cudnn.

Thank you guys. 

After I have installed cuda12.3+cudnn8.9 ( I am using TF 2.16 )and set those environment paths following @Zhaopudark 's guide, GPU can be successfully detected under wsl2(Ubuntun 22.04).

Hope this ticket helps those who are looking for answers. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/63341">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/63341">No</a>
 i install cuda 12.3.2 and cudnn .then when i install tensorflow via `pip install tensorflow[and-cuda]` will auto install`tensorflow 2.16.1` and can not find my GPU. when i input `pip install tensorflow==2.15` . It will autoremove 2.16.1 and install 2.15.then I can find my GPU
 ```
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
```
bash output
```
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
2024-03-13 02:41:48.797545: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-13 02:41:48.797600: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-13 02:41:48.798179: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-13 02:41:48.801510: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-13 02:41:49.293727: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-13 02:41:49.828549: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:08:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-13 02:41:49.934333: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:08:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-13 02:41:49.934403: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:08:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
``` I am stuck with the same issue while upgrading to Tensorflow 2.16 within an Anaconda environment, with an hope to get started with Keras 3.0 released by @fchollet 

I am using Ubuntu on WSL2 and have never set any path variables explicitly starting from TF 2.15 on different machine

**Starting from TF 2.15, this process was automated, but again we are back (at least for the moment)**

TensorFlow team might've missed that setting where they set path when the virtual environment is activated, original motivation was to have different version of CUDA and cuDNN installed in different virtual environments for different TF versions.

I am hopeful this will be fixed soon @tensorflower-gardener  **Listing when just downgraded to TF 2.15 in same environment**
<pre>
(py3) chaudhary@LAPTOP:~$ python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
2024-03-14 17:54:21.028849: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-14 17:54:21.028973: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-14 17:54:21.029714: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-14 17:54:21.034104: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-14 17:54:21.620019: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-14 17:54:23.291920: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-14 17:54:23.317718: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-14 17:54:23.317830: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
</pre>


**The GPU device cannot be detected however for Tensorflow 2.16**
<pre>
(py3) chaudhary@LAPTOP:~$ python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
2024-03-14 17:49:42.014773: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-14 17:49:42.753522: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-14 17:49:43.943419: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-14 17:49:43.943633: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]
</pre>
 hi guys, this is what works for me.

Installation summary, following build configs https://www.tensorflow.org/install/source#gpu
Wsl distro : Ubuntu-22.04
Miniconda with python 3.10

Cuda toolkit 12.3 [cuda](https://developer.nvidia.com/cuda-12-3-2-download-archive?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0&target_type=deb_local)

Cudnn 8.9
use older version, cudnn 8.9 [archive](https://developer.nvidia.com/rdp/cudnn-archive)
follow commands replace with 8.9 version : [cmds](https://developer.nvidia.com/cudnn-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=22.04&target_type=deb_local)

Added these to .bashrc, and source it
```
export PATH=/usr/local/cuda-12.3/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.3/lib64:$LD_LIBRARY_PATH
```

video reference : https://www.youtube.com/watch?v=VE5OiQSfPLg

result:
```
2024-03-17 10:34:59.899645: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-17 10:35:00.567105: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-17 10:35:01.338148: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:07:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-17 10:35:01.393266: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:07:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-17 10:35:01.393375: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:07:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
```

If you think it too much of hassle, i would recommend to use [tensorflow/tensorflow:latest-gpu](https://hub.docker.com/layers/tensorflow/tensorflow/latest-gpu/images/sha256-4ab9ffddd6ffacc9251ac6439f431eb38d66200d3f52397b5d77f9bc3298c4e9?context=explore) . 
I tried both ways, realized container way train the same model faster(likely due hardware configs) and setup time is quick with no issue. @chaudharyachint08. This weekend, I have shared my steps in this [post](https://little-train.com/posts/4f6afcf2.html). Maybe you can check it and transfer it to your own environment. Thanks @zjm008 and @Zhaopudark for providing with workarounds. I certainly agree that these are ways to mitigate this problem
<br><br>

However, while following the [pip-installation guide](https://www.tensorflow.org/install/pip) I can see that
<pre>
pip install tensorflow[and-cuda]
</pre>
do download many cuda libraries including cuDNN as well, which was set to path variables as soon as you activate a conda environment which is a very nice functionality up to TF 2.15

I believe the developers might've missed the same while moving to TF 2.16

Do let me know if you two agree? @chaudharyachint08 I agree with you. Actually, this should not occur, we, users should not need to install cuda and cudnn libraries twice. So, there must be some potential mistakes on optional dependency `[and-cuda]`. I have done some ablation tests before, and I find even though I use conda to install cuda and manually install cudnn to my conda env, the optional dependency `[and-cuda]` is still needed when installing TensorFlow-GPU.  > @chaudharyachint08. This weekend, I have shared my steps in this [post](https://little-train.com/posts/4f6afcf2.html). Maybe you can check it and transfer it to your own environment.

@Zhaopudark  Wow, thank you so much for this! # Almost final and automated fix below

* Where I found the resolution
    * [TF 2.16.1 Fails to work with GPUs](https://github.com/tensorflow/tensorflow/issues/63362)
        * Solution proposed by "sh-shahrokhi", improved by "ChristofKaufmann"
        * See specially [Comment by COntributor](https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1990133275)
    * Related Issues
        * [GPU not detected on WSL2](https://github.com/tensorflow/tensorflow/issues/63341), where I have post some comments
        * [Tensorflow WSL GPU CUDA recognition issue RTX3090](https://github.com/tensorflow/tensorflow/issues/63948)
        * [Once gain: tf.2.16.1 fails to recognize GPUs](https://github.com/keras-team/keras/issues/19276)
    * Other mention on social media
        * [Reddit issue](https://www.reddit.com/r/bashonubuntuonwindows/comments/1bb0xqg/your_kernel_may_have_been_built_without_numa/)

* Exact solution
    * Temporary fix (after activating environment in which Tensorflow 2.16.1 is installed)
        <pre>
        export NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
        export LD_LIBRARY_PATH=$(echo ${NVIDIA_DIR}/*/lib/ | sed -r 's/\s+/:/g')${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
        </pre>
    * Automating the variable set/unset process with Anaconda (one-time setup)
        * Activate your environement in which TF 2.16.1 is installed
        * Two files to be created in "anaconda3/envs/<ENV_NAME>/etc/conda"
            * anaconda3/envs/<ENV_NAME>/etc/conda/activate.d/env_vars.sh
            <pre>
            #!/bin/sh
            export NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
            export LD_LIBRARY_PATH=$(echo ${NVIDIA_DIR}/*/lib/ | sed -r 's/\s+/:/g')${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
            </pre>
            * anaconda3/envs/<ENV_NAME>/etc/conda/deactivate.d/env_vars.sh
            <pre>
            #!/bin/sh
            unset NVIDIA_DIR
            unset LD_LIBRARY_PATH
            </pre>
        * Official documentation to do this via [conda.io](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#macos-and-linux)
        * Stack-overflow question where I got this [Set environment vars when activating conda env](https://stackoverflow.com/questions/31598963/how-to-set-specific-environment-variables-when-activating-conda-environment)


* What else helped me
    * [How to list all variables names and their current values?](https://askubuntu.com/questions/275965/how-to-list-all-variables-names-and-their-current-values)
    * [Conda documenation for "env_vars.sh" for activate/deactivate](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#macos-and-linux)

### Issue type

Build/Install

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

2.16.1

### Custom code

No

### OS platform and distribution

WSL2 Ubuntu 22.04

### Mobile device

_No response_

### Python version

3.11

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

Hi,

I am trying to install run keras3 on wsl2.  But the output of device listing indicates that there is still some dynamic library missing.

I followed the [doc](https://www.tensorflow.org/install/pip#step-by-step_instructions) to finish the installation 

I am using the latest NV driver, version of which can be find on nvidia-smi output.

I have managed to install CUDA 11.8 and cudnn 8.6 on my device which are the exact the same version listed on doc.

Asides from installing cudnn using [Local Installer for Ubuntu22.04 x86_64 (Deb)](https://developer.nvidia.com/compute/cudnn/secure/8.6.0/local_installers/11.8/cudnn-local-repo-ubuntu2204-8.6.0.163_1.0-1_amd64.deb) on [this page](https://developer.nvidia.com/rdp/cudnn-archive), I have also tried to manually copy the cudnn header files and lib files to cuda-11.8/include and cuda-11.8/lib and the  LD_LIBRARY_PATH is updated  according to this [artical](https://discuss.tensorflow.org/t/tensorflow-on-wsl2-cannot-find-my-gpu-despite-following-the-guide-for-tensorflow2-12/15906) and this [one](https://dev.to/_jhohannes/installing-tensorflowwith-gpu-support-in-linux-or-wsl-2le5)

Anything I missed in my steps?

Regards
Sichao Hu

### Standalone code to reproduce the issue

```shell
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
```


### Relevant log output

```shell
TF output:
2024-03-09 16:22:45.531685: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-03-09 16:22:45.787261: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-09 16:22:46.518153: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-09 16:22:47.403086: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-09 16:22:47.474378: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]


This is the output of nvidia-smi:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.60.01              Driver Version: 551.76         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4070 ...    On  |   00000000:01:00.0  On |                  N/A |
|  0%   44C    P8             18W /  285W |    1125MiB /  16376MiB |      5%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+


This is the output of nvcc -V:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
```
