TF 2.16.1 Fails to work with GPUs
It does not work with python=3.12.2 either. Same error. installed tensorflow with `$ pip install tensorflow[and-cuda]` The same error on bare Ubuntu and WSL2 2.15 works without any problems with python 3.11 I have the same problem with **Ubuntu 22.04.4** with the following environment:
- `tensorflow==2.16.1`
- `Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] on linux`
- cuDNN 8.6.0.163
- `gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0`

`nvcc --version` output:
```sh
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
``` I'm not sure if this is the root cause, but I resolved my own issue which also surfaced as a "Cannot dlopen some GPU libraries." error when trying to run `python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"`

To resolve my issue, I followed the tested build versions here:
https://www.tensorflow.org/install/source#gpu

and I needed to update my existing installations from cuDNN 9 -> 8.9 and CUDA 12.4->12.3

When you're on an [NVIDIA download page like this one for CUDA Toolkit](https://developer.nvidia.com/cuda-downloads), don't just download the latest version. See previous versions by hitting "Archive of Previous CUDA Releases"

@JuanVargas can you try uninstalling your existing CUDA installation to a tested build configuration for TF 2.16 by downgrading to CUDA 12.3?

I followed this post to uninstall my existing cuda installation:
https://askubuntu.com/questions/530043/removing-nvidia-cuda-toolkit-and-installing-new-one

@DiegoMont can you try upgrading your cuDNN to 8.9 and CUDA to 12.3? I am having the same issue. Brand new Ubuntu 22.04 WSL2 image. Blank Conda environment with either `python 3.12.*` or `3.11.*` fails to correctly setup tensorflow for GPU use when following the recommended:
`pip install tensorflow[and-cuda]`

Trying to list the physical devices results in:
```
2024-03-11 02:00:00.294704: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-11 02:00:00.709325: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-11 02:00:01.180225: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:2d:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 02:00:01.180445: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]
```
```
cuDNN 8.9.*
Cuda 12.3
Tensorflow 2.16.1
TensorRT 8.6.1
```

Is this a new issue caused by the fact that it doesn't appear that any system cuda needs to be separately installed in WSL2 anymore. I certainly didn't install one manually and yet `nvidia-smi` is happily reporting cuda version 12.3. It probably comes down to some env paths not set correctly but playing around with $CUDA_PATH and guessing the location within the conda environment has not resolved anything. TensorRT doesn't seem to be picked up yet is definitely installed in the conda environment. Pytorch GPU visibility works as expected. Hi @JuanVargas ,

For GPU package you need to ensure the installation of CUDA driver which can be verified with nvidia-smi command. Then you need to install TF-cuda package with `pip install tensorflow[and-cuda]` which automatically installs required cuda/cudnn libraries.

I have checked in colab and able to detect GPU.Please refer attached [gist](https://colab.sandbox.google.com/gist/SuryanarayanaY/808553205834b3043b619980c9ae3ffc/63362.ipynb). 
doublequotes in pip install because of ZSH 

```
pip install "tensorflow[and-cuda]==2.16.1"                                                                       
 

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: tensorflow==2.16.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (2.16.1)
Requirement already satisfied: absl-py>=1.0.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (2.1.0)
Requirement already satisfied: astunparse>=1.6.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (1.6.3)
Requirement already satisfied: flatbuffers>=23.5.26 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (24.3.7)
Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.5.4)
Requirement already satisfied: google-pasta>=0.1.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.2.0)
Requirement already satisfied: h5py>=3.10.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (3.10.0)
Requirement already satisfied: libclang>=13.0.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (16.0.6)
Requirement already satisfied: ml-dtypes~=0.3.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.3.2)
Requirement already satisfied: opt-einsum>=2.3.2 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (3.3.0)
Requirement already satisfied: packaging in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (24.0)
Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (4.25.3)
Requirement already satisfied: requests<3,>=2.21.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (2.31.0)
Requirement already satisfied: setuptools in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (69.1.1)
Requirement already satisfied: six>=1.12.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (1.16.0)
Requirement already satisfied: termcolor>=1.1.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (2.4.0)
Requirement already satisfied: typing-extensions>=3.6.6 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (4.10.0)
Requirement already satisfied: wrapt>=1.11.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (1.16.0)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (1.62.1)
Requirement already satisfied: tensorboard<2.17,>=2.16 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (2.16.2)
Requirement already satisfied: keras>=3.0.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (3.0.5)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.36.0)
Requirement already satisfied: numpy<2.0.0,>=1.23.5 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (1.26.4)
Requirement already satisfied: nvidia-cublas-cu12==12.3.4.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (12.3.4.1)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.3.101 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (12.3.101)
Requirement already satisfied: nvidia-cuda-nvcc-cu12==12.3.107 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (12.3.107)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.3.107 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (12.3.107)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.3.101 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (12.3.101)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.7.29 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (8.9.7.29)
Requirement already satisfied: nvidia-cufft-cu12==11.0.12.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (11.0.12.1)
Requirement already satisfied: nvidia-curand-cu12==10.3.4.107 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (10.3.4.107)
Requirement already satisfied: nvidia-cusolver-cu12==11.5.4.101 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (11.5.4.101)
Requirement already satisfied: nvidia-cusparse-cu12==12.2.0.103 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (12.2.0.103)
Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (2.19.3)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.3.101 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (12.3.101)
Requirement already satisfied: wheel<1.0,>=0.23.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from astunparse>=1.6.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.42.0)
Requirement already satisfied: rich in ./miniconda3/envs/tf/lib/python3.11/site-packages (from keras>=3.0.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (13.7.1)
Requirement already satisfied: namex in ./miniconda3/envs/tf/lib/python3.11/site-packages (from keras>=3.0.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.0.7)
Requirement already satisfied: dm-tree in ./miniconda3/envs/tf/lib/python3.11/site-packages (from keras>=3.0.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.1.8)
Requirement already satisfied: charset-normalizer<4,>=2 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from requests<3,>=2.21.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from requests<3,>=2.21.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from requests<3,>=2.21.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from requests<3,>=2.21.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (2024.2.2)
Requirement already satisfied: markdown>=2.6.8 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorboard<2.17,>=2.16->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (3.5.2)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorboard<2.17,>=2.16->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorboard<2.17,>=2.16->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (3.0.1)
Requirement already satisfied: MarkupSafe>=2.1.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from werkzeug>=1.0.1->tensorboard<2.17,>=2.16->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (2.1.5)
Requirement already satisfied: markdown-it-py>=2.2.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from rich->keras>=3.0.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from rich->keras>=3.0.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (2.17.2)
Requirement already satisfied: mdurl~=0.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich->keras>=3.0.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.1.2)
```


```
nvidia-smi             
                                                                                           
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.60.01              Driver Version: 551.76         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4070 Ti     On  |   00000000:01:00.0  On |                  N/A |
|  0%   39C    P5             10W /  285W |    4334MiB /  12282MiB |     13%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A        41      G   /Xwayland                                   N/A      |
+-----------------------------------------------------------------------------------------+

```

```
python3

Python 3.11.7 (main, Dec 15 2023, 18:12:31) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
print(tf.config.list_physical_devices('GPU'))2024-03-11 09:36:29.601060: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-03-11 09:36:29.921637: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-11 09:36:30.793353: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
>>> print(tf.config.list_physical_devices('GPU'))
2024-03-11 09:36:33.878560: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 09:36:33.980099: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]
>>>
``` ```
nvcc -V 
                                                                                                          
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Feb_27_16:19:38_PST_2024
Cuda compilation tools, release 12.4, V12.4.99
Build cuda_12.4.r12.4/compiler.33961263_0
``` got it work :) first 
https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89-------------------------------- 

then download [Local Installer for Ubuntu22.04 x86_64 (Deb)](https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.7/local_installers/12.x/cudnn-local-repo-ubuntu2204-8.9.7.29_1.0-1_amd64.deb/)  

unpack and install libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb  



```
sudo dpkg -i libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb   
                                                           
Selecting previously unselected package libcudnn8.
(Reading database ... 47318 files and directories currently installed.)
Preparing to unpack libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb ...
Unpacking libcudnn8 (8.9.7.29-1+cuda12.2) ...
Setting up libcudnn8 (8.9.7.29-1+cuda12.2) ...

```


```
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"  

                             
2024-03-11 10:27:47.879686: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-03-11 10:27:47.909157: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-11 10:27:48.316717: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-11 10:27:48.664469: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 10:27:48.688059: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 10:27:48.688111: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

``` Hi Krzysztof

I visited the  site
https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------

where I found an entry listed as " Local Installer for UBuntu22.04
x86_64(Deb)" which I downloaded.
Unfortunately what I got is a package named
"cudnn-local-repo-ubuntu2204-8.9.7.29_1.0-1_amd64.deb"
which is not the same as the name you suggest in your message, which is "
libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb"

I assume what you meant is to get the libcudnn8_8.9.7.29*amd64.deb and
the cuda12.2_amd64.deb separately and install both.

I have CUDA 12.4. I will not go back to trying to make TF 2.16.1 work with
older versions of CUDA (12.2 or 12.3) because sooner or later
the TF team will have to produce a version with the updated version of
CUDA. IMHO, rather than us wasting time going back in versions,
the TF beak should invest time going forward to update TF to the current
CUDA version.

Thank you, Juan


On Mon, Mar 11, 2024 at 5:30 AM Krzysztof Radzikowski <
***@***.***> wrote:

> got it work :) first
>
> https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
>
> then download Local Installer for Ubuntu22.04 x86_64 (Deb)
> <https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.7/local_installers/12.x/cudnn-local-repo-ubuntu2204-8.9.7.29_1.0-1_amd64.deb/>
>
> unpack and install libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb
>
> `
>
> sudo dpkg -i libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb
> Selecting previously unselected package libcudnn8.
> (Reading database ... 47318 files and directories currently installed.)
> Preparing to unpack libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb ...
> Unpacking libcudnn8 (8.9.7.29-1+cuda12.2) ...
> Setting up libcudnn8 (8.9.7.29-1+cuda12.2) ...
>
> `
>
> python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
>
>
> 2024-03-11 10:27:47.879686: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
> 2024-03-11 10:27:47.909157: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
> To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
> 2024-03-11 10:27:48.316717: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
> 2024-03-11 10:27:48.664469: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
> Your kernel may have been built without NUMA support.
> 2024-03-11 10:27:48.688059: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
> Your kernel may have been built without NUMA support.
> 2024-03-11 10:27:48.688111: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
> Your kernel may have been built without NUMA support.
> [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1987970615>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAGK34PV2IIF5FUZ73EPKOTYXV2SZAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXHE3TANRRGU>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
 It's just tensorflow can't see the Cuda libraries.

Instal tensorflow[and-cuda] and add this to your .bashrc or conda activation script. Adjust python version in it according to your setup.

NVIDIA_PACKAGE_DIR="$CONDA_PREFIX/lib/python3.12/site-packages/nvidia"

for dir in $NVIDIA_PACKAGE_DIR/*; do
    if [ -d "$dir/lib" ]; then
        export LD_LIBRARY_PATH="$dir/lib:$LD_LIBRARY_PATH"
    fi
done


You won't need to install cuda or cudnn on the system. only the cuda libraries that are installed with  $ pip install tensorflow[and-cuda] would be enough.


On Mon, Mar 11, 2024, 7:04 a.m. Juan E. Vargas ***@***.***>
wrote:

> Hi Krzysztof
>
> I visited the site
>
> https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
>
> where I found an entry listed as " Local Installer for UBuntu22.04
> x86_64(Deb)" which I downloaded.
> Unfortunately what I got is a package named
> "cudnn-local-repo-ubuntu2204-8.9.7.29_1.0-1_amd64.deb"
> which is not the same as the name you suggest in your message, which is "
> libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb"
>
> I assume what you meant is to get the libcudnn8_8.9.7.29*amd64.deb and
> the cuda12.2_amd64.deb separately and install both.
>
> I have CUDA 12.4. I will not go back to trying to make TF 2.16.1 work with
> older versions of CUDA (12.2 or 12.3) because sooner or later
> the TF team will have to produce a version with the updated version of
> CUDA. IMHO, rather than us wasting time going back in versions,
> the TF beak should invest time going forward to update TF to the current
> CUDA version.
>
> Thank you, Juan
>
>
> On Mon, Mar 11, 2024 at 5:30 AM Krzysztof Radzikowski <
> ***@***.***> wrote:
>
> > got it work :) first
> >
> >
> https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
> >
> > then download Local Installer for Ubuntu22.04 x86_64 (Deb)
> > <
> https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.7/local_installers/12.x/cudnn-local-repo-ubuntu2204-8.9.7.29_1.0-1_amd64.deb/>
>
> >
> > unpack and install libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb
> >
> > `
> >
> > sudo dpkg -i libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb
> > Selecting previously unselected package libcudnn8.
> > (Reading database ... 47318 files and directories currently installed.)
> > Preparing to unpack libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb ...
> > Unpacking libcudnn8 (8.9.7.29-1+cuda12.2) ...
> > Setting up libcudnn8 (8.9.7.29-1+cuda12.2) ...
> >
> > `
> >
> > python3 -c "import tensorflow as tf;
> print(tf.config.list_physical_devices('GPU'))"
> >
> >
> > 2024-03-11 10:27:47.879686: I tensorflow/core/util/port.cc:113] oneDNN
> custom operations are on. You may see slightly different numerical results
> due to floating-point round-off errors from different computation orders.
> To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
> > 2024-03-11 10:27:47.909157: I
> tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary
> is optimized to use available CPU instructions in performance-critical
> operations.
> > To enable the following instructions: AVX2 AVX_VNNI FMA, in other
> operations, rebuild TensorFlow with the appropriate compiler flags.
> > 2024-03-11 10:27:48.316717: W
> tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could
> not find TensorRT
> > 2024-03-11 10:27:48.664469: I
> external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not
> open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
> > Your kernel may have been built without NUMA support.
> > 2024-03-11 10:27:48.688059: I
> external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not
> open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
> > Your kernel may have been built without NUMA support.
> > 2024-03-11 10:27:48.688111: I
> external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not
> open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
> > Your kernel may have been built without NUMA support.
> > [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
> >
> >
> > —
> > Reply to this email directly, view it on GitHub
> > <
> https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1987970615>,
>
> > or unsubscribe
> > <
> https://github.com/notifications/unsubscribe-auth/AAGK34PV2IIF5FUZ73EPKOTYXV2SZAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXHE3TANRRGU>
>
> > .
> > You are receiving this because you were mentioned.Message ID:
> > ***@***.***>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1988394624>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AZRPJAGFJU5ZGBGHOSUK6DTYXWTU3AVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBYGM4TINRSGQ>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
 will try that and will let you know. Thank you for the suggestion. Juan


On Mon, Mar 11, 2024 at 10:52 AM Shayan Shahrokhi ***@***.***>
wrote:

> It's just tensorflow can't see the Cuda libraries.
>
> Instal tensorflow[and-cuda] and add this to your .bashrc or conda
> activation script
>
> NVIDIA_PACKAGE_DIR="$CONDA_PREFIX/lib/python3.12/site-packages/nvidia"
>
> for dir in $NVIDIA_PACKAGE_DIR/*; do
> if [ -d "$dir/lib" ]; then
> export LD_LIBRARY_PATH="$dir/lib:$LD_LIBRARY_PATH"
> fi
> done
>
>
>
> On Mon, Mar 11, 2024, 7:04 a.m. Juan E. Vargas ***@***.***>
> wrote:
>
> > Hi Krzysztof
> >
> > I visited the site
> >
> >
> https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
> >
> > where I found an entry listed as " Local Installer for UBuntu22.04
> > x86_64(Deb)" which I downloaded.
> > Unfortunately what I got is a package named
> > "cudnn-local-repo-ubuntu2204-8.9.7.29_1.0-1_amd64.deb"
> > which is not the same as the name you suggest in your message, which is
> "
> > libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb"
> >
> > I assume what you meant is to get the libcudnn8_8.9.7.29*amd64.deb and
> > the cuda12.2_amd64.deb separately and install both.
> >
> > I have CUDA 12.4. I will not go back to trying to make TF 2.16.1 work
> with
> > older versions of CUDA (12.2 or 12.3) because sooner or later
> > the TF team will have to produce a version with the updated version of
> > CUDA. IMHO, rather than us wasting time going back in versions,
> > the TF beak should invest time going forward to update TF to the current
> > CUDA version.
> >
> > Thank you, Juan
> >
> >
> > On Mon, Mar 11, 2024 at 5:30 AM Krzysztof Radzikowski <
> > ***@***.***> wrote:
> >
> > > got it work :) first
> > >
> > >
> >
> https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
> > >
> > > then download Local Installer for Ubuntu22.04 x86_64 (Deb)
> > > <
> >
> https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.7/local_installers/12.x/cudnn-local-repo-ubuntu2204-8.9.7.29_1.0-1_amd64.deb/>
>
> >
> > >
> > > unpack and install libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb
> > >
> > > `
> > >
> > > sudo dpkg -i libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb
> > > Selecting previously unselected package libcudnn8.
> > > (Reading database ... 47318 files and directories currently
> installed.)
> > > Preparing to unpack libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb ...
> > > Unpacking libcudnn8 (8.9.7.29-1+cuda12.2) ...
> > > Setting up libcudnn8 (8.9.7.29-1+cuda12.2) ...
> > >
> > > `
> > >
> > > python3 -c "import tensorflow as tf;
> > print(tf.config.list_physical_devices('GPU'))"
> > >
> > >
> > > 2024-03-11 10:27:47.879686: I tensorflow/core/util/port.cc:113] oneDNN
> > custom operations are on. You may see slightly different numerical
> results
> > due to floating-point round-off errors from different computation
> orders.
> > To turn them off, set the environment variable
> `TF_ENABLE_ONEDNN_OPTS=0`.
> > > 2024-03-11 10:27:47.909157: I
> > tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow
> binary
> > is optimized to use available CPU instructions in performance-critical
> > operations.
> > > To enable the following instructions: AVX2 AVX_VNNI FMA, in other
> > operations, rebuild TensorFlow with the appropriate compiler flags.
> > > 2024-03-11 10:27:48.316717: W
> > tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning:
> Could
> > not find TensorRT
> > > 2024-03-11 10:27:48.664469: I
> > external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could
> not
> > open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
> > > Your kernel may have been built without NUMA support.
> > > 2024-03-11 10:27:48.688059: I
> > external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could
> not
> > open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
> > > Your kernel may have been built without NUMA support.
> > > 2024-03-11 10:27:48.688111: I
> > external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could
> not
> > open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
> > > Your kernel may have been built without NUMA support.
> > > [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
> > >
> > >
> > > —
> > > Reply to this email directly, view it on GitHub
> > > <
> >
> https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1987970615>,
>
> >
> > > or unsubscribe
> > > <
> >
> https://github.com/notifications/unsubscribe-auth/AAGK34PV2IIF5FUZ73EPKOTYXV2SZAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXHE3TANRRGU>
>
> >
> > > .
> > > You are receiving this because you were mentioned.Message ID:
> > > ***@***.***>
> > >
> >
> > —
> > Reply to this email directly, view it on GitHub
> > <
> https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1988394624>,
>
> > or unsubscribe
> > <
> https://github.com/notifications/unsubscribe-auth/AZRPJAGFJU5ZGBGHOSUK6DTYXWTU3AVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBYGM4TINRSGQ>
>
> > .
> > You are receiving this because you commented.Message ID:
> > ***@***.***>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1988630226>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAGK34PLSBZFKQ5AQKIZ7X3YXXAMRAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBYGYZTAMRSGY>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
 Hi Shayan Shahrokhi

Thank you for your suggestion  (adding the location of the site-packages. I
hope you would not mind if I ask :

I saw that in your suggestion the name python 3.12 is listed. Is that the
version of python that you used to test TF 2.16.1 compatibility with CUDA?

Thank you, Juan



On Mon, Mar 11, 2024 at 11:01 AM Juan Vargas ***@***.***> wrote:

> will try that and will let you know. Thank you for the suggestion. Juan
>
>
> On Mon, Mar 11, 2024 at 10:52 AM Shayan Shahrokhi <
> ***@***.***> wrote:
>
>> It's just tensorflow can't see the Cuda libraries.
>>
>> Instal tensorflow[and-cuda] and add this to your .bashrc or conda
>> activation script
>>
>> NVIDIA_PACKAGE_DIR="$CONDA_PREFIX/lib/python3.12/site-packages/nvidia"
>>
>> for dir in $NVIDIA_PACKAGE_DIR/*; do
>> if [ -d "$dir/lib" ]; then
>> export LD_LIBRARY_PATH="$dir/lib:$LD_LIBRARY_PATH"
>> fi
>> done
>>
>>
>>
>> On Mon, Mar 11, 2024, 7:04 a.m. Juan E. Vargas ***@***.***>
>> wrote:
>>
>> > Hi Krzysztof
>> >
>> > I visited the site
>> >
>> >
>> https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
>> >
>> > where I found an entry listed as " Local Installer for UBuntu22.04
>> > x86_64(Deb)" which I downloaded.
>> > Unfortunately what I got is a package named
>> > "cudnn-local-repo-ubuntu2204-8.9.7.29_1.0-1_amd64.deb"
>> > which is not the same as the name you suggest in your message, which is
>> "
>> > libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb"
>> >
>> > I assume what you meant is to get the libcudnn8_8.9.7.29*amd64.deb and
>> > the cuda12.2_amd64.deb separately and install both.
>> >
>> > I have CUDA 12.4. I will not go back to trying to make TF 2.16.1 work
>> with
>> > older versions of CUDA (12.2 or 12.3) because sooner or later
>> > the TF team will have to produce a version with the updated version of
>> > CUDA. IMHO, rather than us wasting time going back in versions,
>> > the TF beak should invest time going forward to update TF to the
>> current
>> > CUDA version.
>> >
>> > Thank you, Juan
>> >
>> >
>> > On Mon, Mar 11, 2024 at 5:30 AM Krzysztof Radzikowski <
>> > ***@***.***> wrote:
>> >
>> > > got it work :) first
>> > >
>> > >
>> >
>> https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
>> > >
>> > > then download Local Installer for Ubuntu22.04 x86_64 (Deb)
>> > > <
>> >
>> https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.7/local_installers/12.x/cudnn-local-repo-ubuntu2204-8.9.7.29_1.0-1_amd64.deb/>
>>
>> >
>> > >
>> > > unpack and install libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb
>> > >
>> > > `
>> > >
>> > > sudo dpkg -i libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb
>> > > Selecting previously unselected package libcudnn8.
>> > > (Reading database ... 47318 files and directories currently
>> installed.)
>> > > Preparing to unpack libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb ...
>> > > Unpacking libcudnn8 (8.9.7.29-1+cuda12.2) ...
>> > > Setting up libcudnn8 (8.9.7.29-1+cuda12.2) ...
>> > >
>> > > `
>> > >
>> > > python3 -c "import tensorflow as tf;
>> > print(tf.config.list_physical_devices('GPU'))"
>> > >
>> > >
>> > > 2024-03-11 10:27:47.879686: I tensorflow/core/util/port.cc:113]
>> oneDNN
>> > custom operations are on. You may see slightly different numerical
>> results
>> > due to floating-point round-off errors from different computation
>> orders.
>> > To turn them off, set the environment variable
>> `TF_ENABLE_ONEDNN_OPTS=0`.
>> > > 2024-03-11 10:27:47.909157: I
>> > tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow
>> binary
>> > is optimized to use available CPU instructions in performance-critical
>> > operations.
>> > > To enable the following instructions: AVX2 AVX_VNNI FMA, in other
>> > operations, rebuild TensorFlow with the appropriate compiler flags.
>> > > 2024-03-11 10:27:48.316717: W
>> > tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning:
>> Could
>> > not find TensorRT
>> > > 2024-03-11 10:27:48.664469: I
>> > external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could
>> not
>> > open file to read NUMA node:
>> /sys/bus/pci/devices/0000:01:00.0/numa_node
>> > > Your kernel may have been built without NUMA support.
>> > > 2024-03-11 10:27:48.688059: I
>> > external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could
>> not
>> > open file to read NUMA node:
>> /sys/bus/pci/devices/0000:01:00.0/numa_node
>> > > Your kernel may have been built without NUMA support.
>> > > 2024-03-11 10:27:48.688111: I
>> > external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could
>> not
>> > open file to read NUMA node:
>> /sys/bus/pci/devices/0000:01:00.0/numa_node
>> > > Your kernel may have been built without NUMA support.
>> > > [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
>> > >
>> > >
>> > > —
>> > > Reply to this email directly, view it on GitHub
>> > > <
>> >
>> https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1987970615>,
>>
>> >
>> > > or unsubscribe
>> > > <
>> >
>> https://github.com/notifications/unsubscribe-auth/AAGK34PV2IIF5FUZ73EPKOTYXV2SZAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXHE3TANRRGU>
>>
>> >
>> > > .
>> > > You are receiving this because you were mentioned.Message ID:
>> > > ***@***.***>
>> > >
>> >
>> > —
>> > Reply to this email directly, view it on GitHub
>> > <
>> https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1988394624>,
>>
>> > or unsubscribe
>> > <
>> https://github.com/notifications/unsubscribe-auth/AZRPJAGFJU5ZGBGHOSUK6DTYXWTU3AVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBYGM4TINRSGQ>
>>
>> > .
>> > You are receiving this because you commented.Message ID:
>> > ***@***.***>
>> >
>>
>> —
>> Reply to this email directly, view it on GitHub
>> <https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1988630226>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/AAGK34PLSBZFKQ5AQKIZ7X3YXXAMRAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBYGYZTAMRSGY>
>> .
>> You are receiving this because you were mentioned.Message ID:
>> ***@***.***>
>>
>
 It's the python in the environment that I installed tensorflow[and-cuda]
there.

On Mon, Mar 11, 2024, 9:30 a.m. Juan E. Vargas ***@***.***>
wrote:

> Hi Shayan Shahrokhi
>
> Thank you for your suggestion (adding the location of the site-packages. I
> hope you would not mind if I ask :
>
> I saw that in your suggestion the name python 3.12 is listed. Is that the
> version of python that you used to test TF 2.16.1 compatibility with CUDA?
>
> Thank you, Juan
>
>
>
> On Mon, Mar 11, 2024 at 11:01 AM Juan Vargas ***@***.***> wrote:
>
> > will try that and will let you know. Thank you for the suggestion. Juan
> >
> >
> > On Mon, Mar 11, 2024 at 10:52 AM Shayan Shahrokhi <
> > ***@***.***> wrote:
> >
> >> It's just tensorflow can't see the Cuda libraries.
> >>
> >> Instal tensorflow[and-cuda] and add this to your .bashrc or conda
> >> activation script
> >>
> >> NVIDIA_PACKAGE_DIR="$CONDA_PREFIX/lib/python3.12/site-packages/nvidia"
> >>
> >> for dir in $NVIDIA_PACKAGE_DIR/*; do
> >> if [ -d "$dir/lib" ]; then
> >> export LD_LIBRARY_PATH="$dir/lib:$LD_LIBRARY_PATH"
> >> fi
> >> done
> >>
> >>
> >>
> >> On Mon, Mar 11, 2024, 7:04 a.m. Juan E. Vargas ***@***.***>
> >> wrote:
> >>
> >> > Hi Krzysztof
> >> >
> >> > I visited the site
> >> >
> >> >
> >>
> https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
> >> >
> >> > where I found an entry listed as " Local Installer for UBuntu22.04
> >> > x86_64(Deb)" which I downloaded.
> >> > Unfortunately what I got is a package named
> >> > "cudnn-local-repo-ubuntu2204-8.9.7.29_1.0-1_amd64.deb"
> >> > which is not the same as the name you suggest in your message, which
> is
> >> "
> >> > libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb"
> >> >
> >> > I assume what you meant is to get the libcudnn8_8.9.7.29*amd64.deb
> and
> >> > the cuda12.2_amd64.deb separately and install both.
> >> >
> >> > I have CUDA 12.4. I will not go back to trying to make TF 2.16.1 work
> >> with
> >> > older versions of CUDA (12.2 or 12.3) because sooner or later
> >> > the TF team will have to produce a version with the updated version
> of
> >> > CUDA. IMHO, rather than us wasting time going back in versions,
> >> > the TF beak should invest time going forward to update TF to the
> >> current
> >> > CUDA version.
> >> >
> >> > Thank you, Juan
> >> >
> >> >
> >> > On Mon, Mar 11, 2024 at 5:30 AM Krzysztof Radzikowski <
> >> > ***@***.***> wrote:
> >> >
> >> > > got it work :) first
> >> > >
> >> > >
> >> >
> >>
> https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
> >> > >
> >> > > then download Local Installer for Ubuntu22.04 x86_64 (Deb)
> >> > > <
> >> >
> >>
> https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.7/local_installers/12.x/cudnn-local-repo-ubuntu2204-8.9.7.29_1.0-1_amd64.deb/>
>
> >>
> >> >
> >> > >
> >> > > unpack and install libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb
> >> > >
> >> > > `
> >> > >
> >> > > sudo dpkg -i libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb
> >> > > Selecting previously unselected package libcudnn8.
> >> > > (Reading database ... 47318 files and directories currently
> >> installed.)
> >> > > Preparing to unpack libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb ...
> >> > > Unpacking libcudnn8 (8.9.7.29-1+cuda12.2) ...
> >> > > Setting up libcudnn8 (8.9.7.29-1+cuda12.2) ...
> >> > >
> >> > > `
> >> > >
> >> > > python3 -c "import tensorflow as tf;
> >> > print(tf.config.list_physical_devices('GPU'))"
> >> > >
> >> > >
> >> > > 2024-03-11 10:27:47.879686: I tensorflow/core/util/port.cc:113]
> >> oneDNN
> >> > custom operations are on. You may see slightly different numerical
> >> results
> >> > due to floating-point round-off errors from different computation
> >> orders.
> >> > To turn them off, set the environment variable
> >> `TF_ENABLE_ONEDNN_OPTS=0`.
> >> > > 2024-03-11 10:27:47.909157: I
> >> > tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow
> >> binary
> >> > is optimized to use available CPU instructions in
> performance-critical
> >> > operations.
> >> > > To enable the following instructions: AVX2 AVX_VNNI FMA, in other
> >> > operations, rebuild TensorFlow with the appropriate compiler flags.
> >> > > 2024-03-11 10:27:48.316717: W
> >> > tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning:
> >> Could
> >> > not find TensorRT
> >> > > 2024-03-11 10:27:48.664469: I
> >> > external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984]
> could
> >> not
> >> > open file to read NUMA node:
> >> /sys/bus/pci/devices/0000:01:00.0/numa_node
> >> > > Your kernel may have been built without NUMA support.
> >> > > 2024-03-11 10:27:48.688059: I
> >> > external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984]
> could
> >> not
> >> > open file to read NUMA node:
> >> /sys/bus/pci/devices/0000:01:00.0/numa_node
> >> > > Your kernel may have been built without NUMA support.
> >> > > 2024-03-11 10:27:48.688111: I
> >> > external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984]
> could
> >> not
> >> > open file to read NUMA node:
> >> /sys/bus/pci/devices/0000:01:00.0/numa_node
> >> > > Your kernel may have been built without NUMA support.
> >> > > [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
> >> > >
> >> > >
> >> > > —
> >> > > Reply to this email directly, view it on GitHub
> >> > > <
> >> >
> >>
> https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1987970615>,
>
> >>
> >> >
> >> > > or unsubscribe
> >> > > <
> >> >
> >>
> https://github.com/notifications/unsubscribe-auth/AAGK34PV2IIF5FUZ73EPKOTYXV2SZAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXHE3TANRRGU>
>
> >>
> >> >
> >> > > .
> >> > > You are receiving this because you were mentioned.Message ID:
> >> > > ***@***.***>
> >> > >
> >> >
> >> > —
> >> > Reply to this email directly, view it on GitHub
> >> > <
> >>
> https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1988394624>,
>
> >>
> >> > or unsubscribe
> >> > <
> >>
> https://github.com/notifications/unsubscribe-auth/AZRPJAGFJU5ZGBGHOSUK6DTYXWTU3AVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBYGM4TINRSGQ>
>
> >>
> >> > .
> >> > You are receiving this because you commented.Message ID:
> >> > ***@***.***>
> >> >
> >>
> >> —
> >> Reply to this email directly, view it on GitHub
> >> <
> https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1988630226>,
>
> >> or unsubscribe
> >> <
> https://github.com/notifications/unsubscribe-auth/AAGK34PLSBZFKQ5AQKIZ7X3YXXAMRAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBYGYZTAMRSGY>
>
> >> .
> >> You are receiving this because you were mentioned.Message ID:
> >> ***@***.***>
> >>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1988718350>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AZRPJACFR4YH7UCXMOEH7MTYXXEXVAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBYG4YTQMZVGA>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
 
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/  you can get  .deb file there directrly
 Thanks @sh-shahrokhi. I thought it was path related. Modified slightly to make it python version independent if you put it in your conda environment activation (`[environment]/etc/activate.d/env_vars.sh`).

```
NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
for dir in $NVIDIA_DIR/*; do
    if [ -d "$dir/lib" ]; then
        export LD_LIBRARY_PATH="$dir/lib:$LD_LIBRARY_PATH"
    fi
done
```
This is not a resolution as this post install step should not be necessary.

`W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT`

I can't seem to do similar tricks to resolve the TensorRT issues when installed similarly into the conda environment. Any ideas? > Thanks @sh-shahrokhi. I thought it was path related. Modified slightly to make it python version independent if you put it in your conda environment activation (`[environment]/etc/activate.d/env_vars.sh`).
> 
> ```
> NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
> for dir in $NVIDIA_DIR/*; do
>     if [ -d "$dir/lib" ]; then
>         export LD_LIBRARY_PATH="$dir/lib:$LD_LIBRARY_PATH"
>     fi
> done
> ```
> 
> This is not a resolution as this post install step should not be necessary.
> 
> `W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT`
> 
> I can't seem to do similar tricks to resolve the TensorRT issues when installed similarly into the conda environment. Any ideas?

I don't actually use TensorRT, but I  would check if the required .so file for it is visible to tensorflow. Maybe I would need to find the name of required file in tensorflow source code.

This actually doesn't change the fact that the new tensorflow version should be tested by google team before release, or the bugs should be fixed. It seems they only care about having a working docker image, not anything else.
 I have given up on TensorRT. I guess I won't be using it either.

> This actually doesn't change the fact that the new tensorflow version should be tested by google team before release, or the bugs should be fixed. It seems they only care about having a working docker image, not anything else.

Agreed. Installing TF has always been hit or miss and it seems that in the many years since I last used TF that hasn't changed one bit. 
 Well, I wasted 8hr of my Sunday on this setting up another pc from scratch. Before reverting to the old version. Now looking to move off tensor flow. In general, we used to test RC versions before release. For example, we used to have RC0, RC1 and RC2 for TF 2.9. This gave people and downstream teams enough time to test and report issues.

It seems that 2.16.1 only had an RC0 (for 2.16.0).

The release process is (was?) like this:
- cut the release branch (e.g., `r2.17`)
- immediately trigger the release pipeline. This would create a few PRs to update version numbers, release notes, but after this step RC0 should be as close as possible to the version on master branch at the time the release branch has been cut. There should not be any code changes to the release branch at this point (except to maybe cherrypick fixes from master from hard bugs caused by cutting the branch at a wrong commit)
- have at least a week of testing for downstream teams to test RC0
- get fixes to discovered bugs landed on master, cherrypick them to release branch, after they are already tested on nightly releases
- trigger RC1 pipeline. Again, no other code changes should occur now, except to fix bugs discovered during building
- wait a week for downstream teams to test. If there are bugs, repeat the steps above for another RC, otherwise repeat the steps above for the final version.

Overall, this process would take `number_of_RCs + 1` weeks with a possibility of a few more weeks of delay.

However, for 2.16 release, although the branch was cut on Feb 8th, there has been only one RC. Most likely issues can be solved by a patch release Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/63362">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/63362">No</a>
 I am closing this (unresolved issue) because I am told by the Keras/TF team that the issue is related to TF.  So where is the resolution to this quite annoying installation issue being tracked? 

It’s kind of ridiculous that a fresh install using the official method results in a system that can’t access the GPUs out of the box. As others have noted this sort of dealbreaker when setting up a new machine pushes people away from TF.  I closed the issue because I was asked to do so by a TF/Keras team member,
for the reasons that I stated in the closing comment. Just like you (Leigh)
I am disappointed, and Idecided not to waste more time. There are strong
reasons for some of us (users) to prefer using our own hardware. I hope the
TF/keras people get that message. Thanks, Juan


> Message ID: ***@***.***>
>
 You were asked to close the issue on keras, because it is not a keras issue. This is tensorflow, where proper detection of the cuda libs is definitely still an issue. Please reopen.  I already opened a similar issue in TF.

On Tue, Mar 12, 2024 at 9:20 AM Sam Levang ***@***.***> wrote:

> You were asked to close the issue on keras, because it is not a keras
> issue. This is tensorflow, where proper detection of the cuda libs is
> definitely still an issue. Please reopen.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1991637460>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAGK34OEUIJZCSKQOGZCGQTYX36J7AVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJRGYZTONBWGA>
> .
> You are receiving this because you modified the open/close state.Message
> ID: ***@***.***>
>
 Again, **you are on the tensorflow repo**. You closed the wrong issue. Oh, sorry about that!

On Tue, Mar 12, 2024 at 9:46 AM Sam Levang ***@***.***> wrote:

> Again, *you are on the tensorflow repo*. You closed the wrong issue.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1991690836>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAGK34K46DAULWCUUNQHYKDYX4BLJAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJRGY4TAOBTGY>
> .
> You are receiving this because you modified the open/close state.Message
> ID: ***@***.***>
>
 Issue reopened It is really frustrating indeed. The silly thing is that with JAX as the Keras backend GPU is utilized smoothly (still my RTX3060 GPU is not in the list of physical devices that TensorFlow's framework is aware of (!) ).. I guess we cannot do something else but wait until the next version release... Thank you Mr. Juan for raising the issue here.
### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

TF 2.16.1

### Custom code

No

### OS platform and distribution

Linux Ubuntu 22.04.4 LTS

### Mobile device

_No response_

### Python version

3.10.12

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

12.4

### GPU model and memory

_No response_

### Current behavior?

I created a python venv in which I installed TF 2.16.1 following your instructions: pip install tensorflow
When I run python, import tf,  and issue tf.config.list_physical_devices('GPU') 
I get an empty list  [ ]

I created another python venv, installed TF 2.16.1, only this time with the instructions: 

 python3 -m pip install tensorflow[and-cuda]
 
When I run that version, import tensorflow as tf, and  issue

 tf.config.list_physical_devices('GPU') 

I also get an empty list. 

BTW, I have no problems running  on my box TF 2.15.1 with GPUs. Julia also works just fine with GPUs and so does PyTorch.
the 
 


### Standalone code to reproduce the issue

```shell
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2024-03-09 19:15:45.018171: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-09 19:15:50.412646: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
>>> tf.__version__
'2.16.1'

tf.config.list_physical_devices('GPU') 
2024-03-09 19:16:28.923792: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-03-09 19:16:29.078379: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]
>>>
```


### Relevant log output

_No response_